Solving overdetermined system by QR decomposition












3












$begingroup$


I need to solve $Ax=b$ in lots of ways using QR decomposition.



$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$



This is an overdetermined system. That is, it has more equations than needed for a unique solution.



I need to find $min ||Ax-b||$. How should I solve it using QR?



I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



but what do I do after this?






linear-algebra numerical-methods numerical-linear-algebra







share|cite|improve this question









$endgroup$

















    3












    $begingroup$


    I need to solve $Ax=b$ in lots of ways using QR decomposition.



    $$A = begin{bmatrix}
    1 & 1 \
    -1 & 1 \
    1 & 2
    end{bmatrix}, b = begin{bmatrix}
    1 \
    0 \
    1
    end{bmatrix}$$



    This is an overdetermined system. That is, it has more equations than needed for a unique solution.



    I need to find $min ||Ax-b||$. How should I solve it using QR?



    I know that QR can be used to reduce the problem to
    $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



    but what do I do after this?






    linear-algebra numerical-methods numerical-linear-algebra







    share|cite|improve this question









    $endgroup$















      3












      3








      3





      $begingroup$


      I need to solve $Ax=b$ in lots of ways using QR decomposition.



      $$A = begin{bmatrix}
      1 & 1 \
      -1 & 1 \
      1 & 2
      end{bmatrix}, b = begin{bmatrix}
      1 \
      0 \
      1
      end{bmatrix}$$



      This is an overdetermined system. That is, it has more equations than needed for a unique solution.



      I need to find $min ||Ax-b||$. How should I solve it using QR?



      I know that QR can be used to reduce the problem to
      $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



      but what do I do after this?






      linear-algebra numerical-methods numerical-linear-algebra







      share|cite|improve this question









      $endgroup$




      I need to solve $Ax=b$ in lots of ways using QR decomposition.



      $$A = begin{bmatrix}
      1 & 1 \
      -1 & 1 \
      1 & 2
      end{bmatrix}, b = begin{bmatrix}
      1 \
      0 \
      1
      end{bmatrix}$$



      This is an overdetermined system. That is, it has more equations than needed for a unique solution.



      I need to find $min ||Ax-b||$. How should I solve it using QR?



      I know that QR can be used to reduce the problem to
      $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



      but what do I do after this?






      linear-algebra numerical-methods numerical-linear-algebra




      linear-algebra numerical-methods numerical-linear-algebra






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked 12 hours ago









      Guerlando OCsGuerlando OCs

      9721856




      9721856






















          2 Answers
          2






          active

          oldest

          votes


















          7












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$









          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago





















          2












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$









          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago












          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "69"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          noCode: true, onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          7












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$









          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago


















          7












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$









          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago
















          7












          7








          7





          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$



          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.







          share|cite|improve this answer














          share|cite|improve this answer



          share|cite|improve this answer








          edited 8 hours ago

























          answered 12 hours ago









          IanIan

          69.2k25392




          69.2k25392








          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago
















          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago










          1




          1




          $begingroup$
          Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
          $endgroup$
          – Martin Argerami
          12 hours ago




          $begingroup$
          Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
          $endgroup$
          – Martin Argerami
          12 hours ago












          $begingroup$
          @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
          $endgroup$
          – Ian
          12 hours ago






          $begingroup$
          @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
          $endgroup$
          – Ian
          12 hours ago













          2












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$









          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago
















          2












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$









          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago














          2












          2








          2





          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$



          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$







          share|cite|improve this answer












          share|cite|improve this answer



          share|cite|improve this answer










          answered 12 hours ago









          Adam LatosińskiAdam Latosiński

          5488




          5488








          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago














          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago








          1




          1




          $begingroup$
          The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
          $endgroup$
          – Ian
          7 hours ago




          $begingroup$
          The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
          $endgroup$
          – Ian
          7 hours ago


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Mathematics Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          Use MathJax to format equations. MathJax reference.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          Last logged in always never, not logging

          Image
          0 I turned on logging module, it works, but no logs in log_customer or log_visitor. Those tables are empty. What else can block logging? Magento 1.9 magento-1.9 magento-1 share asked 4 mins ago G Strato G Strato 43 6 add a comment  |  0 I turned on logging
          Read more

          Colouring column values based on a specific condition. How could I do this?

          Image
          1 I am generating a big-table from a csv file using the latex code as given below. documentclass{article} usepackage{booktabs} usepackage{csvsimple} usepackage{caption} usepackage{longtable} begin{document} csvautobooklongtable[ table head={% toprule bfseries Name &bfseries ID \ midrule}, table foot={\bottomrule}]% {sample-data.csv} end{document} The generated document looks similar to: Now, I want to make: IDs greater than 6 in bold and red. How could I do this? Also, for IDs less than 6, I want to append a symbol < . So 3 becomes 3 < . How could I do this? Here is link to the CSV file that I am using to create a table. color longtable conditionals
          Read more

          Iĥnotaksono

          Image
          Iĥnogena Thalassinoides : nestoj produktitaj de krustuloj el meza Ĵurasio, Maĥteŝ Katan, suda Israelo. Iĥnotaksono estas difinita de la Internacia Kodo de Zoologia Nomenklaturo kiel "taksono bazita sur fosiliigita laboro de organismo", tio estas, la ne-homa ekvivalento de arkeologia objekto. Iĥnotaksonoj estas termino uzata por restoj kiuj utilas por identigi kaj distingi morfologie distingajn iĥnofosiliojn, pli komune konataj kiel spuraj fosilioj. Oni atribuas al ili rangojn de genro kaj specio fare de iĥnologistoj, multe kiel ĉe organismoj ĉe la Linea taksonomio. Tiuj estas konataj kiel iĥnogenroj kaj iĥnospecioj , respektive. Iĥnotaksonoj inkludas spurajn fosiliojn kiaj nestotruoj, truoj kaj gravuraĵojs, spuroj kaj spurvojoj, koprolitoj, gastrolitoj, regurgitolitoj, nestoj, insektaj minvojoj, mordoj kaj ungaj strukturoj, sekrecioj modifitaj de organisma agado, kiaj pupajl ujoj, araneretoj, kaj plantaj gajloj. Iĥnotaksono derivas el la greka ίχνος, iĥnos sign
          Read more