Skip to content

Instantly share code, notes, and snippets.

@nfitch

nfitch/mdb_demo.md

Last active August 3, 2016 02:13
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save nfitch/cdfac5547fb72270c2f7 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save nfitch/cdfac5547fb72270c2f7 to your computer and use it in GitHub Desktop.
Download ZIP
mdb_demo.md
Raw
mdb_demo.md

External Sources

Using MDB

A full guide should be coming out in the near future, but for now you can build node from source, which will generate a mdb_v8.so:

[root@dev-1 ~/node-0.12/node-v0.12.0]# find . -name mdb_v8.so
./out/Release/mdb_v8.so

This is what you want to ::load to get the latest debugging tools for 0.12. (::jsconstructor, ::jssource, ::jsfunctions, etc.). For example:

[root@dev-1 ~]# ls /root/node-0.12/node-v0.12.0/out/Release/mdb_v8.so
/root/node-0.12/node-v0.12.0/out/Release/mdb_v8.so
[root@dev-1 ~/linux_node]# ls
core  node
[root@dev-1 ~/linux_node]# mdb ./node ./core
mdb: warning: librtld_db failed to initialize; shared library information will not be available
> ::load /root/node-0.12/node-v0.12.0/out/Release/mdb_v8.so
V8 version: 3.14.5.9
Autoconfigured V8 support from target
C++ symbol demangling enabled
> ::jsstack
native: v8::internal::OS::Abort+0xe
native: v8::internal::Isolate::DoThrow+0x3ce
native: v8::internal::Isolate::Throw+9
native: v8::internal::Runtime_Throw+0x3d
        (1 internal frame elided)
js:     increment
js:     processImmediate
        (1 internal frame elided)
        (1 internal frame elided)
native: _ZN2v88internalL6InvokeEbNS0_6HandleINS0_10JSFunctionEEENS1_INS0...

Some mdb examples

::jsstack

Helpful when the node app blows up or is 'frozen'. For example:

Old version:

fffffd7fffdf8570 libc.so.1`_lwp_kill+0xa
fffffd7fffdf85a0 libc.so.1`raise+0x20
fffffd7fffdf85f0 libc.so.1`abort+0x98
fffffd7fffdf8600 0x6d10e9
fffffd7fffdf8700
_ZN2v88internalL21Builtin_HandleApiCallENS0_12_GLOBAL__N_116BuiltinArgumentsILNS0_21BuiltinExtraArgum
entsE1EEEPNS0_7IsolateE+0x1a3
fffffd7fffdf8728 0x5e8d81206362 internal (Code: 5e8d812062c1)
fffffd7fffdf8758 0x5e8d8143c7dc respond (4a651f844681)
fffffd7fffdf87b0 0x5e8d8143c481 next (4ba7c3e30841)
...

New Version:

> ::jsstack
native: libc.so.1`_lwp_kill+0xa
native: libc.so.1`raise+0x20
native: libc.so.1`abort+0x98
        (1 internal frame elided)
native: _ZN2v88internalL21Builtin_HandleApiCallENS0_12_GLOBAL__N_116Buil...
        (1 internal frame elided)
js:     respond
js:     next
        (1 internal frame elided)
        (1 internal frame elided)
...

::findjsobjects

> ::findjsobjects
          OBJECT #OBJECTS   #PROPS CONSTRUCTOR: PROPS
      7a55cf21b531        1        0 MathConstructor
      5768550fcbb9        1        0 Router
      7a55cf2073f1        1        0 JSON
      601f1d7b8069        1        0 Domain
...
> ::findjsobjects -p total
4282f854bbe1
> ::findjsobjects -p total | ::findjsobjects | ::jsprint
{
    "success": 33560575,
    "total": <unknown JavaScript object type "Map">,
}
{
    "success": 1591,
    "total": 1591,
}
{
    "success": 0,
    "total": 0,
}

Finding JS leaks

> ::findjsobjects ! sort -n -k 2 | tail -20
# Then walk up until you find the object that is holding on
> ::findjsobjects ! sort -n -k 2 | tail -1
    457a07a041a9    11158        0 Array
> 457a07a041a9::jsprint
[]
> 457a07a041a9::findjsobjects -r
457a07a041a9 referred to by 47c8732dd761.buffer
# Look at the object and see if there might be something "weird"
> 47c8732dd761::jsprint
# Check that there are "many" of what you're looking at
> 47c8732dd761::findjsobjects ! wc -l
2027
# Continue up the stack
> 47c8732dd761::findjsobjects -r
47c8732dd761 referred to by 47c8732b66c1._readableState
> 47c8732b66c1::findjsobjects -r
47c8732b66c1 referred to by 5768550c9501[1252]
> 5768550c9501::findjsobjects -r
5768550c9501 referred to by 5768550cf471.conns
> 5768550cf471::jsprint conns ! grep destroyed | sort | uniq -c
      1         "destroyed": false,
     1591         "destroyed": true,

Getting cores from Linux hosts

A "full" guide is under the link in External Sources, but the gist is:

First get a core dumped on the Linux host:

# Make sure it is 64 bit
linux$ Linux ubuntu 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
linux$ ulimit -c unlimited
linux$ node --abort-on-uncaught-exception die.js
...
Trace/breakpoint trap (core dumped)
linux$ which node
/home/nate/.nvm/v0.10.36/bin/node

Now from the smartos host:

smartos$ uname -a
SunOS smartos 5.11 joyent_20150123T200224Z i86pc i386 i86pc Solaris
smartos$ scp linux:/home/nate/.nvm/v0.10.36/bin/node .
smartos$ scp linux:~/core .
smartos$ mdb ./node ./core
mdb: warning: librtld_db failed to initialize; shared library information will not be available
> ::load v8
V8 version: 3.14.5.9
Autoconfigured V8 support from target
C++ symbol demangling enabled
> ::jsstack
7fff91619ba0 v8::internal::OS::Abort+0xe
7fff91619c50 v8::internal::Isolate::DoThrow+0x3ce
...

Finding the remote address for a socket

An interesting exercise was my attempting to track down the tcp address for a connection. Looking at the object, there was nothing that looked like it should have a handle to the remote ip address. However, I knew that a socket does have a remoteAddress property:

http://ww.nodejs.org/api/net.html#net_socket_remoteaddress

So I looked at the node.js source:

https://github.com/joyent/node/blob/v0.12.0-release/lib/net.js#L584

Socket.prototype._getpeername = function() {
  if (!this._handle || !this._handle.getpeername) {
    return {};
  }
  if (!this._peername) {
    var out = {};
    var err = this._handle.getpeername(out);
    if (err) return {}; // FIXME(bnoordhuis) Throw?
    this._peername = out;
  }
  return this._peername;
};

Socket.prototype.__defineGetter__('remoteAddress', function() {
  return this._getpeername().address;
});

Note how the remoteAddress defines a getter, which calls out to _getpeername, which gets the address from this._handle.getpeername. Looking in that same file for where _handle is set:

this._handle = pipe ? createPipe() : createTCP();
...
function createTCP() {
  var TCP = process.binding('tcp_wrap').TCP;
  return new TCP();
}

So that is a TcpWrap object, which is in the cpp bindings:

https://github.com/joyent/node/blob/v0.12.0-release/src/tcp_wrap.cc#L143

NODE_SET_PROTOTYPE_METHOD(t, "getpeername", GetPeerName);
...
int err = uv_tcp_getpeername(&wrap->handle_,
                             reinterpret_cast<sockaddr*>(&address),
                             &addrlen);
...
const sockaddr* addr = reinterpret_cast<const sockaddr*>(&address);
AddressToJS(env, addr, out);

Which delves down into lib uv's guts. I don't know of a way to get at that data via mdb (though there may be some way I'm not aware of), it's interesting that the JS layer caches the result (cutting down the function above):

Socket.prototype._getpeername = function() {
  if (!this._peername) {
    var out = {};
    var err = this._handle.getpeername(out);
    if (err) return {}; // FIXME(bnoordhuis) Throw?
    // Caches in _peername!!!
    this._peername = out;
  }
  return this._peername;
};

From there it was simple just to access the property on server connection, which gives the Socket object a _peername:

Code:

server.on('connection', function (socket) {
  socket.remoteAddress;
});

mdb result:

> 496d7cdda361::jsprint -a conns
496d7cde07d9: [
    496d7cc0f5e9: {
        "_connecting": 561220904161: false,
...
> 496d7cc0f5e9::jsprint _peername
{
    "address": "127.0.0.1",
    "family": "IPv4",
    "port": 41960,
}

It would be nice to know how to get the address "directly".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment