Hyperbolic spaces have emerged as an effective manifold to learn representations due to their ability to efficiently represent hierarchical data structures, with little distortion, even for low-dimensional embeddings. In the chosen hyperbolic model, such as the Poincaré ball, classification is usually conducted by leveraging a signed distance function to the hyperbolic equivalent of a plane (gyroplanes) or by measuring the alignment to a virtual fixed prototype. We propose, in a deep learning context, to leverage a different characterization of a decision boundary: Horospheres, which are level-sets of the Busemann function. They are geometrically equivalent to spheres tangent to the boundary of the hyperbolic space on a virtual point akin to a prototype. Accordingly, we define a new horospherical layer that can be adapted to any neural network backbone. In previous works, prototypes are usually uniformly distributed without using a potentially available label hierarchy for the task at hand. We also propose a hierarchically informed method for positioning these prototypes, based on the Gromov-Wasserstein distance. We find that the combination of a good initialization and optimization of the prototypes improves the baseline performance for image classification on hierarchical datasets and in two semantic segmentation tasks, conducted on image and point cloud datasets. Source code will be released upon acceptance.