Showing content from http://mail.python.org/pipermail/python-dev/attachments/20160114/718f888d/attachment-0001.html below:
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Light";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Monaco;
panose-1:2 11 5 9 3 4 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:2127655759;
mso-list-type:hybrid;
mso-list-template-ids:-1447677676 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Gflags/umdh is pretty useful on Windows, I used it to track down a few quirky PyParallel memory leaks. Steps:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Enable global flags: gflags âi python.exe +ust<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Launch Python.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Enable the umdh tracer: umdh âp:<python.exe pid> -f:d1.log<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Kill it after a short run.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">5.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Re-launch Python.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">6.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Enable it again: umdh âp:<python.exe pid> -f:d2.log<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">7.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Let it run for longer â long enough to make sure itâs leaking memory.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><span style="mso-list:Ignore">8.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Kill it, then generate a dump file: umdh âd d1.log d2.log > dump.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">(Those steps were pretty specific to my particular situation, but it should at least be a reasonable starting point for what to google to find out more.)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Here are two sample outputs that pin-pointed the exact leak path:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">+ 49116 ( 49116 - 0)
<span style="background:yellow;mso-highlight:yellow">6 allocs</span> BackTrace9763CA0<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">+ 6 ( 6 - 0) BackTrace9763CA0 allocations<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!RtlpCallInterceptRoutine+40<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!RtlAllocateHeap+79846<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">SQLSRV32!SQLAllocateMemory+26<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">markSQLSRV32!SQLAllocConnect+F6<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">SQLSRV32!SQLAllocHandle+83<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!RetcodeDriverInit+2D9<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!SQLInternalDriverConnectW+2F<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!CDispenser::CreateResource+DB<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">comsvcs!CHolder::SafeDispenserDriver::CreateResource+43<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">comsvcs!CHolder::AllocResource+24D<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!CDispenser::TryAllocResource+6E<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!CDispenser::GetActiveConnection+72<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!SQLDriverConnectW+9D4<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">+ 48432 ( 48432 - 0)
<span style="background:yellow;mso-highlight:yellow">6 allocs</span> BackTrace97635E0<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">+ 6 ( 6 - 0) BackTrace97635E0 allocations<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!RtlpCallInterceptRoutine+40<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!RtlAllocateHeap+79846<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">SQLSRV32!SQLAllocateMemory+26<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">SQLSRV32!SQLAllocConnect+4D<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">SQLSRV32!SQLAllocHandle+83<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!RetcodeDriverInit+2D9<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!SQLInternalDriverConnectW+2F<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!CDispenser::CreateResource+DB<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">comsvcs!CHolder::SafeDispenserDriver::CreateResource+43<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">comsvcs!CHolder::AllocResource+24D<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!CDispenser::TryAllocResource+6E<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!CDispenser::GetActiveConnection+72<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ODBC32!SQLDriverConnectW+9D4<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Segoe UI Light",sans-serif;color:black"> <o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Segoe UI Light",sans-serif;color:black">You can see that the leak is coming from SQLAllocateMemory in this case, which we trace back to pyodbc Connection_New(). In this
particular case, we were continually creating new connections, but never deallocating them.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Segoe UI Light",sans-serif">Another example:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Segoe UI Light",sans-serif"><o:p> </o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">+ 9420800 ( 9420800 - 0)
<span style="background:yellow;mso-highlight:yellow">2300 allocs</span> BackTrace8E96BE0<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">+ 2300 ( 2300 - 0) BackTrace8E96BE0 allocations<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:27.0pt;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!RtlpCallInterceptRoutine+40<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!RtlAllocateHeap+79846<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!_PyTLSHeap_Init+9E (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 3040)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!_PyTLSHeap_Malloc+B0 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 3124)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!_PyHeap_Malloc+55 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 3170)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyUnicode_New+149 (c:\users\trent\home\src\pyparallel\objects\unicodeobject.c, 1051)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyUnicode_DecodeUTF8Stateful+D2 (c:\users\trent\home\src\pyparallel\objects\unicodeobject.c, 4910)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyUnicode_InternFromString+16 (c:\users\trent\home\src\pyparallel\objects\unicodeobject.c, 14505)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_GetAttrString+2E (c:\users\trent\home\src\pyparallel\objects\object.c, 881)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!lowercase+18 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.h, 69)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!execute+772 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\cursor.cpp, 869)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">pyodbc!Cursor_execute+D4 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\cursor.cpp, 945)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyCFunction_Call+12D (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 117)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">python33!PxSocket_IOCallback+295 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 10612)<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">KERNEL32!BasepTpIoCallback+59<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!TppIopExecuteCallback+182<o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.75in;margin-bottom:.0001pt">
<span style="font-size:8.0pt;font-family:Monaco;color:black">ntdll!TppWorkerThread+8B4<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Segoe UI Light",sans-serif;color:black"> <o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Segoe UI Light",sans-serif;color:black">We can trace this leak back to lowercase(), which was interning a string. At this point, we were intercepting string interning and
making it thread local allocation, but those allocations were never being freed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"> Python-Dev [mailto:python-dev-bounces+trent=snakebite.org@python.org]
<b>On Behalf Of </b>Matthew Paulson<br>
<b>Sent:</b> Wednesday, January 13, 2016 7:18 PM<br>
<b>To:</b> Andrew Barnert <abarnert@yahoo.com><br>
<b>Cc:</b> Python Dev <python-dev@python.org>; Gary F. Desrochers <desrochers@busiq.com><br>
<b>Subject:</b> Re: [Python-Dev] Discussion related to memory leaks requested<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Andrew:<br>
<br>
These are all good points, and I defer to your experience -- I am new to python internals, but the fact remains that after multiple iterations of our embedded test case, we are seeing continued allocations (DS2015) and growth of the working set (windows task
manager). If your are pooling resources on the free list, wouldn't you expect these items to get reused and for things to stabilize after a while? We're not seeing that.<br>
<br>
I think Victor's suggestion of a very simple test case is probably the best idea. I'll try to put that together in the next few days and if it also demonstrates the problem, then I'll submit it here.<br>
<br>
Thanks for your time and help.<br>
<br>
Best,<br>
<br>
Matt<o:p></o:p></p>
<div>
<p class="MsoNormal">On 1/13/2016 6:45 PM, Andrew Barnert wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Jan 13, 2016, at 14:49, Matthew Paulson <<a href="mailto:paulson@busiq.com">paulson@busiq.com</a>> wrote:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Hi Victor:<br>
<br>
No, I'm using the new heap analysis functions in DS2015. <o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Isn't that going to report any memory that Python's higher level allocators hold in their freelists as leaked, even though it isn't leaked?<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">We think we have found one issue. In the following sequence, dict has no side effects, yet it is used -- unless someone can shed light on why dict is used in this case:<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Where do you see an issue here? The dict will have one ref, so the decref at the end should return it to the freelist.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Also, it looks like there _is_ a side effect here. When you add a bunch of elements to a dict, it grows. When you delete a bunch of elements, it generally doesn't shrink. But when you clear the dict, it does shrink. So, copying it to a
temporary dict, clearing it, updating it from the temporary dict, and then releasing the temporary dict should force it to shrink.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So, the overall effect should be that you have a smaller hash table for the builtins dict, and a chunk of memory sitting on the freelists ready to be reused. If your analyzer is showing the freelists as leaked, this will look like a net
leak rather than a net recovery, but that's just a problem in the analyzer.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Of course I could be wrong, but I think the first step is to rule out the possibility that you're measuring the wrong thing...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">/* Clear the modules dict. */<br>
PyDict_Clear(modules);<br>
/* Restore the original builtins dict, to ensure that any<br>
user data gets cleared. */<br>
dict = PyDict_Copy(interp->builtins);<br>
if (dict == NULL)<br>
PyErr_Clear();<br>
PyDict_Clear(interp->builtins);<br>
if (PyDict_Update(interp->builtins, interp->builtins_copy))<br>
PyErr_Clear();<br>
Py_XDECREF(dict);<br>
<br>
And removing dict from this sequence seems to have fixed one of the issues, yielding 14k per iteration.<o:p></o:p></p>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Simple program: Good idea. We will try that -- right now it's embedded in a more complex environment, but we have tried to strip it down to a very simple sequence.<br>
<br>
The next item on our list is memory that is not getting freed after running simple string. It's in the parsertok sequence -- it seems that the syntax tree is not getting cleared -- but this opinion is preliminary.<br>
<br>
Best,<br>
<br>
Matt<o:p></o:p></p>
<div>
<p class="MsoNormal">On 1/13/2016 5:10 PM, Victor Stinner wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Hi,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>2016-01-13 20:32 GMT+01:00 Matthew Paulson <a href="mailto:paulson@busiq.com"><paulson@busiq.com></a>:<o:p></o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>I've spent some time performing memory leak analysis while using Python in an embedded configuration.<o:p></o:p></pre>
</blockquote>
<pre>Hum, did you try tracemalloc?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><a href="https://docs.python.org/dev/library/tracemalloc.html">https://docs.python.org/dev/library/tracemalloc.html</a><o:p></o:p></pre>
<pre><a href="https://pytracemalloc.readthedocs.org/">https://pytracemalloc.readthedocs.org/</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Is there someone in the group that would like to discuss this topic. There seems to be other leaks as well. I'm new to Python-dev, but willing to help or work with someone who is more familiar with these areas than I.<o:p></o:p></pre>
</blockquote>
<pre>Are you able to reproduce the leak with a simple program?<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Victor<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">-- <br>
<MattSig.JPG><o:p></o:p></p>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com">
https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com</a><o:p></o:p></p>
</div>
</blockquote>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">-- <br>
<img border="0" width="887" height="275" id="_x0000_i1025" src="cid:image001.jpg@01D14E51.93AED0C0"><o:p></o:p></p>
</div>
</div>
</body>
</html>
RetroSearch is an open source project built by @garambo
| Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4