Skip to content

Instantly share code, notes, and snippets.

@dancsiqueira
Last active December 14, 2015 17:37
Show Gist options
  • Select an option

  • Save dancsiqueira/7aeb929ab61b1f1c2a9d to your computer and use it in GitHub Desktop.

Select an option

Save dancsiqueira/7aeb929ab61b1f1c2a9d to your computer and use it in GitHub Desktop.
  • Agrupando por tipo de produto (ProductName) e para cada tipo, retorna os tipos de uso (UsageType), o custo (BlendedCost) e as operações (Operation) efetuadas.
  • O resultado final ($out) fica armazenado em uma nova collection chamada billing_product_usage
db.billing.aggregate( [
  { $match : { PayerAccountId: 264381961167 }},
  { $group : { _id:  "$ProductName", usageParams: { $push: { usageType: "$UsageType", cost: "$BlendedCost", operation: "$Operation" } } } },
  { $out: "billing_product_usage" }], { allowDiskUse: true }
)

{
  "_id" : "AWS Key Management Service",
  "usageParams" : [
    {
      "type" : "us-west-1-KMS-Requests",
      "cost" : 0,
      "operation" : "ListKeys"
    },
    {
      "type" : "eu-central-1-KMS-Requests",
      "cost" : 0,
      "operation" : "ListKeys"
    },
    {
      "type" : "eu-central-1-KMS-Requests",
      "cost" : 0,
      "operation" : "ListAliases"
    },
    {
      "type" : "us-west-1-KMS-Requests",
      "cost" : 0,
      "operation" : "ListAliases"
    }
  ]
}
  • Agrupando os tipos de uso iguais de um mesmo produto, somando os custos e aninhando as operações executadas em um array de operações.
  • O resultado final ($out) fica armazenado em uma nova collection chamada billing_product_tag_operation
db.billing_product_usage.aggregate([
  { $unwind: "$usageParams" },
  { "$group": { "_id": { "id": "$_id", "type": "$usageParams.usageType", "operation":"$usageParams.operation" }, "cost": { "$sum": "$usageParams.cost" }}},
  { "$group": { "_id": "$_id.id", "values": { "$push": { "type": "$_id.type", "cost": "$cost", "operation":"$_id.operation" }}}},
  { "$out": "billing_product_tag"}
])

db.billing_product_tag.aggregate([
  {$unwind: "$values"},
  {"$group": {"_id": { "id": "$_id", "type":"$values.type"}, "cost":{ "$sum":"$values.cost"}, "operation":{ "$push":  "$values.operation"}}},
  {"$group": {"_id": "$_id.id", value: { "$push": { "type":"$_id.type", "cost": "$cost", "operation":"$operation"}}}},
  {"$out": "billing_product_tag_operation"}
])

{
  "_id" : "AWS Key Management Service",
  "value" : [
    {
      "type" : "us-west-1-KMS-Requests",
      "cost" : 0,
      "operation" : [
        "ListKeys",
        "ListAliases"
      ]
    },
    {
      "type" : "eu-central-1-KMS-Requests",
      "cost" : 0,
      "operation" : [
        "ListKeys",
        "ListAliases"
      ]
    }
  ]
}

@rgiaviti
Copy link

Danilo, você chegou a testar com uma quantidade massiva de dados no Mongo? Digo isso porque fui testar aqui com uma carga de 1.4GB de dados no Mongo e o aggregate não se deu bem com o $out:

db.billing.aggregate( [
...   { $match : { PayerAccountId: 264381961167 }},
...   { $group : { _id:  "$ProductName", usageParams: { $push: { usageType: "$UsageType", cost: "$BlendedCost", operation: "$Operation" } } } },
...   { $out: "billing_product_usage" }], { allowDiskUse: true }
... )
assert: command failed: {
    "errmsg" : "exception: BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit.",
    "code" : 13548,
    "ok" : 0
} : aggregate failed
Error: command failed: {
    "errmsg" : "exception: BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit.",
    "code" : 13548,
    "ok" : 0
} : aggregate failed
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at Function.assert.commandWorked (src/mongo/shell/assert.js:254:5)
    at DBCollection.aggregate (src/mongo/shell/collection.js:1278:12)
    at (shell):1:12
2015-12-14T15:32:42.767-0200 E QUERY    Error: command failed: {
    "errmsg" : "exception: BufBuilder attempted to grow() to 134217728 bytes, past the 64MB limit.",
    "code" : 13548,
    "ok" : 0
} : aggregate failed
    at Error (<anonymous>)
    at doassert (src/mongo/shell/assert.js:11:14)
    at Function.assert.commandWorked (src/mongo/shell/assert.js:254:5)
    at DBCollection.aggregate (src/mongo/shell/collection.js:1278:12)
    at (shell):1:12 at src/mongo/shell/assert.js:13

Existem algumas coisas na web a respeito sobre esse erro no aggregate, mas parece que o $out combinado com o aggregate em quantidade massiva não se dão bem. Eu achava que era configuração. Mas parece não ser. A princípio. https://www.google.com.br/webhp?sourceid=chrome-instant&ion=1&espv=2&es_th=1&ie=UTF-8#q=mongo+past+the+64MB+limit&es_th=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment