Solving overdetermined system by QR decomposition












3












$begingroup$


I need to solve $Ax=b$ in lots of ways using QR decomposition.



$$A = begin{bmatrix}
1 & 1 \
-1 & 1 \
1 & 2
end{bmatrix}, b = begin{bmatrix}
1 \
0 \
1
end{bmatrix}$$



This is an overdetermined system. That is, it has more equations than needed for a unique solution.



I need to find $min ||Ax-b||$. How should I solve it using QR?



I know that QR can be used to reduce the problem to
$$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



but what do I do after this?






linear-algebra numerical-methods numerical-linear-algebra







share|cite|improve this question









$endgroup$

















    3












    $begingroup$


    I need to solve $Ax=b$ in lots of ways using QR decomposition.



    $$A = begin{bmatrix}
    1 & 1 \
    -1 & 1 \
    1 & 2
    end{bmatrix}, b = begin{bmatrix}
    1 \
    0 \
    1
    end{bmatrix}$$



    This is an overdetermined system. That is, it has more equations than needed for a unique solution.



    I need to find $min ||Ax-b||$. How should I solve it using QR?



    I know that QR can be used to reduce the problem to
    $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



    but what do I do after this?






    linear-algebra numerical-methods numerical-linear-algebra







    share|cite|improve this question









    $endgroup$















      3












      3








      3





      $begingroup$


      I need to solve $Ax=b$ in lots of ways using QR decomposition.



      $$A = begin{bmatrix}
      1 & 1 \
      -1 & 1 \
      1 & 2
      end{bmatrix}, b = begin{bmatrix}
      1 \
      0 \
      1
      end{bmatrix}$$



      This is an overdetermined system. That is, it has more equations than needed for a unique solution.



      I need to find $min ||Ax-b||$. How should I solve it using QR?



      I know that QR can be used to reduce the problem to
      $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



      but what do I do after this?






      linear-algebra numerical-methods numerical-linear-algebra







      share|cite|improve this question









      $endgroup$




      I need to solve $Ax=b$ in lots of ways using QR decomposition.



      $$A = begin{bmatrix}
      1 & 1 \
      -1 & 1 \
      1 & 2
      end{bmatrix}, b = begin{bmatrix}
      1 \
      0 \
      1
      end{bmatrix}$$



      This is an overdetermined system. That is, it has more equations than needed for a unique solution.



      I need to find $min ||Ax-b||$. How should I solve it using QR?



      I know that QR can be used to reduce the problem to
      $$Vert Ax - b Vert = Vert QRx - b Vert = Vert Rx - Q^{-1}b Vert.$$



      but what do I do after this?






      linear-algebra numerical-methods numerical-linear-algebra




      linear-algebra numerical-methods numerical-linear-algebra






      share|cite|improve this question













      share|cite|improve this question











      share|cite|improve this question




      share|cite|improve this question










      asked 12 hours ago









      Guerlando OCsGuerlando OCs

      9721856




      9721856






















          2 Answers
          2






          active

          oldest

          votes


















          7












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$









          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago





















          2












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$









          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago












          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "69"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          noCode: true, onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          7












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$









          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago


















          7












          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$









          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago
















          7












          7








          7





          $begingroup$

          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.






          share|cite|improve this answer











          $endgroup$



          The most straightforward way I know is to pass through the normal equations:



          $$A^T A x = A^T b$$



          and substitute in the $QR$ decomposition of $A$ (with the convention $Q in mathbb{R}^{m times n},R in mathbb{R}^{n times n}$). Thus you get



          $$R^T Q^T Q R x = R^T Q^T b.$$



          But $Q^T Q=I_n$. (Note that in this convention $Q$ isn't an orthogonal matrix, so $Q Q^T neq I_m$, but this doesn't matter here.) Thus:



          $$R^T R x = R^T Q^T b.$$



          If $A$ has linearly independent columns (as is usually the case with overdetermined systems), then $R^T$ is injective, so by multiplying both sides by the left inverse of $R^T$ you get



          $$Rx=Q^T b.$$



          This system is now easy to solve numerically.



          For numerical purposes it's important that the removal of $Q^T Q$ and $R^T$ from the problem is done analytically, and in particular $A^T A$ is never constructed numerically.







          share|cite|improve this answer














          share|cite|improve this answer



          share|cite|improve this answer








          edited 8 hours ago

























          answered 12 hours ago









          IanIan

          69.2k25392




          69.2k25392








          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago
















          • 1




            $begingroup$
            Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
            $endgroup$
            – Martin Argerami
            12 hours ago










          • $begingroup$
            @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
            $endgroup$
            – Ian
            12 hours ago










          1




          1




          $begingroup$
          Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
          $endgroup$
          – Martin Argerami
          12 hours ago




          $begingroup$
          Is there any reason to make this so convoluted? From $Ax=b$ you have $QRx=b$, multiply by $Q^T$ on the left.
          $endgroup$
          – Martin Argerami
          12 hours ago












          $begingroup$
          @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
          $endgroup$
          – Ian
          12 hours ago






          $begingroup$
          @MartinArgerami Because actually the least squares solution usually does not satisfy $Ax=b$. This simple perspective only shows you that this approach gives you a solution when a solution exists. Now you could argue directly that multiplying both sides by $Q^T$ furnishes an equation whose solution is the least squares solution. (Such an argument would resemble the usual geometric argument for deriving the normal equations.) This would make a good alternative answer to mine.
          $endgroup$
          – Ian
          12 hours ago













          2












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$









          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago
















          2












          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$









          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago














          2












          2








          2





          $begingroup$

          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$






          share|cite|improve this answer









          $endgroup$



          Note that $Rx$ has the form
          $$Rx = begin{bmatrix} y_1 \ y_2 \ 0end{bmatrix} $$
          , so if $$ Q^{-1}b = begin{bmatrix} z_1 \ z_2 \ z_3end{bmatrix}$$
          then $|| Rx - Q^{-1}b||$ will be minimal for $y_1 = z_1$, $y_2=z_2$. This set of equation is no longer overdetermined.



          Using matrix notation, if tou write $R = begin{bmatrix} R_1 \ 0end{bmatrix}$ and intoduce $P=begin{bmatrix}1 & 0 & 0 \ 0 & 1& 0end{bmatrix}$, then you have
          $$ R_1x = PQ^{-1}b$$
          $$ x = (R_1)^{-1}PQ^{-1}b$$







          share|cite|improve this answer












          share|cite|improve this answer



          share|cite|improve this answer










          answered 12 hours ago









          Adam LatosińskiAdam Latosiński

          5488




          5488








          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago














          • 1




            $begingroup$
            The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
            $endgroup$
            – Ian
            7 hours ago








          1




          1




          $begingroup$
          The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
          $endgroup$
          – Ian
          7 hours ago




          $begingroup$
          The key trick in this answer is that by the orthogonality, $| Ax - b | = | Rx - Q^T b |$.
          $endgroup$
          – Ian
          7 hours ago


















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Mathematics Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          Use MathJax to format equations. MathJax reference.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3185239%2fsolving-overdetermined-system-by-qr-decomposition%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          What other Star Trek series did the main TNG cast show up in?

          Image
          .everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0; } 4 I'm referring to Picard, Riker, Data, Worf, Geordi, Dr. Crusher, Wesley Crusher, Chief O'Brien and family, Deanna Troi, Guinan. I know Picard and Worf show up on DS9, though I haven't seen it myself. I'm curious what happened to the characters after that last episode of TNG? star-trek star-trek-tng share | improve this question edited 10 hours ago Ham Sandwich 6,051 2 25 75
          Read more

          Berlina muro

          Image
          La berlina muro ĉe Bethaniendamm, 1986 La berlina muro aŭ simple la muro [1] , nomita ankaŭ "kontraŭfaŝisma remparo" en propaganda lingvo de la iama Germana Demokratia Respubliko, estis parto de la interngermana limo, kiu dividis de la 13-a de aŭgusto 1961 ĝis la 9-a de novembro 1989 okcidentan Berlinon de la orienta parto de la urbo kaj de la ĉirkaŭa tereno de la GDR. Ĝi estis unu el la plej famaj simboloj de la malvarma milito kaj de la disdivido de Germanio. Pli ol simpla muro, la berlina muro estis fakte tuta defendokonstruaĵo kun du muroj, ena malplena strio, rondirvojo, gardturoj kaj alarmsistemoj. Dum la provo transiri la severe garditan limon al okcidenta Berlino multaj homoj estis mortigitaj. La preciza nombro de viktimoj estas pridisputita kaj ne certa; la nombroj anoncataj varias inter 86 kaj 238 mortintoj. La malplifortiĝo de Sovetunio kaj la politiko de liberigo gvidita de Miĥail Gorbaĉov ebligis al la orientgermanoj faligi la 9-an de novembro 1989 l
          Read more

          Berlina aerponto

          Image
          Flugkoridoroj dum la blokado de Berlino La Berlina aerponto estas la provizigo de okcidenta Berlino per aviadiloj fare de la aliancanoj inter la 23-a de junio 1948 kaj la 12-a de majo 1949 dum la blokado de Berlino fare de la sovetia armeo. Enhavo 1 Historio 1.1 La aerkoridoroj 1.2 Unuaj flugoj 1.3 Plibonigo de la aerponto 1.4 Fino de la aerponto 1.5 Rozinbombiloj 1.6 Bilanco 2 Flughavenoj 2.1 Berlino 2.2 Okcidenta Germanio 3 Aviadiloj de la berlina aerponto 4 15-a datreveno de la aerponto 5 Bibliografio 6 Referencoj 7 Eksteraj ligiloj Historio | General Lucius D. Clay, militguberniestro de la usona okupadzono Jam antaŭ la berlina aerponto estiĝis la malgranda aerponto . La estro de la sovetia militadministracio en Germanio, Vasili Daniloviĉ Sokolovski, ordonis la 1-an de aprilo 1948, kiel reago al la konferenco de Londono inter britoj, francoj kaj reprezentantoj de Benelukso pri aligo de Germa
          Read more