About your algorithm. It can return both a vector along its greater and lesser heights. In addition, the number of points on different sides of the line can be different, which can lead to quite large errors.
I would do something like the following. I will describe without formulas, choose them yourself to achieve the result.
- Draw a rectangle, the sides of which lie along the axes of coordinates and pass through at least one point. That is, it turns out that something like a rectangle described around the points with two sides parallel to the X axis and two to the Y axis.

- The points that lie on the sides of the resulting rectangle will be the vertices of the new quadrilateral. If there are sides on which there is more than one point, then we build all quadrilaterals that succeed (with the condition that there is only one vertex of the new rectangle on one side of the rectangle). After that, from the resulting quadrangles, choose the only one, the modulus of the difference in the lengths of the diagonals of which will be the smallest, and the average value of any two opposite corners will be as close as possible to 90 degrees.

- In the resulting quadrilateral, from each of its vertices, we build perpendiculars to two sides that do not belong to this vertex. We have 8 intersection points of the resulting perpendiculars with straight lines passing through the sides of the quadrilateral (2 at each vertex).

- Between the two points near each vertex, obtained in the previous paragraph, we draw segments.
- We put dots on the centers of these segments.

- These points will be the vertices of the final quadrilateral. Next you know.

Threat If it is too difficult, I can draw pictures, but really do not want. Read the algorithm and draw the pictures yourself; it will become clearer on paper.
ZYY Kartinochki added. As you can see, the resulting rectangle is not exactly in the place where you would spend it and, possibly, different in size, but this is not important to you. The main thing is that the direction and proportions are preserved, which is enough to get the desired result.
However, if you need not only the direction of the vector, but also its size, you can make inscribed and described rectangles from the final rectangle (without rotating the final one, but simply by changing the size and position). Next, connect their nearest vertices with segments and build a rectangle through the centers of these segments. But, as I understand it, this is not required.
ZYYY Formulas that you need are quite simple from the initial geometry courses.
ZYYYY This decision is based on intuition and I do not want to prove his loyalty now.