【elasticsearch】关于elasticsearch的max_result_window限制问题的解决方式思考

事情起因：我们使用es作为日志搜索引擎，客户收集到的业务日志非常之大，每次查询后，返回页数较多，由于我们web界面限制每页返回150条，当客户翻到66页之后就会报错。

文章目录

前言

二、实验

1.默认生成20条数据

2.默认查询

3.Search after查询

4.Search after二次查询

总结

前言

报错信息如下：

Elasticsearch limits the search result to 10000 messages. With a page size of 150 messages, you can use the first 66 pages. Unable to perform search query: Elasticsearch exception [type=illegal_argument_exception, reason=Result window is too large, from + size must be less than or equal to: [10000] but was [34050]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.].

一、Search after介绍

使用Search after作为查询，默认情况下我们使用Scroll search作为查询，可以指定From + size 大小的数据批量返回，这是为了防止因为请求过多的结果而导致内存和CPU资源过度消耗。

查询结果的分页可以通过使用from and size 来完成，但是当达到深度分页时，成本变得令人望而却步。默认为 10000 是一种保护措施，搜索请求占用的堆内存和时间成正比。建议使用 scroll API 来实现高效的深度滚动，但滚动上下文成本高昂，而且成本不高，建议将其用于实时用户请求。 Search after通过提供实时游标来规避此问题。这个想法是使用上一页的结果来帮助检索下一页。

二、实验

1.默认生成20条数据

for i in `seq 1 20`
do 
    echo 192.168.113.195 -d -P 54000 "zhongguoren $i"
    sleep 1
done

2.默认查询

GET index_1/_search
{
  "query": {
    "match": {
      "message": "zhongguoren"
    }
  },
  "sort": [
    {
      "timestamp": "asc"
    }
  ]
}

默认返回20条数据，通过默认查询知识确认，每条信息的流入时间和条数，目前按照1-20进行排序，没有重复的时间戳。

3.Search after查询

index_1/_search
{
  "size": 10,
  "query": {
    "match": {
      "message": "zhongguoren"
    }
  },
  "sort": [
    {
      "timestamp": "asc"
    },
    {
      "_id": "asc"
    }
  ]
}

先查询前10条，这个和之前默认查询有个区别，就是排序时通过timestamp和_id进行同时排序，在查询结果中得到一个sort字段，值为

"sort": [ -
          1680770417895,
          "a894bf71-d456-11ed-aa52-000c29f6b211"
        ]

查询后结果值为：

{ - 
  "took": 223,
  "timed_out": false,
  "_shards": { - 
    "total": 4,
    "successful": 4,
    "skipped": 0,
    "failed": 0
  },
  "hits": { - 
    "total": { - 
      "value": 20,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [ - 
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "2df06a80-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 35834,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVC899ZZXDJXF453F0RVXJ",
          "message": "<5>Apr 6 16:36:52 root: zhongguoren 1",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:36:52.135"
        },
        "sort": [ - 
          1680770212135,
          "2df06a80-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "5244ccf0-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 52885,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVE3XCWZK71D74YVF58PZ2",
          "message": "<5>Apr 6 16:37:53 root: zhongguoren 2",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:37:53.086"
        },
        "sort": [ - 
          1680770273086,
          "5244ccf0-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "5e46d610-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 57417,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVEQFJP3CN17A2NG41YYHY",
          "message": "<5>Apr 6 16:38:13 root: zhongguoren 3",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:38:13.232"
        },
        "sort": [ - 
          1680770293232,
          "5e46d610-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "644eb870-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 52483,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVF1BQNJZ48GW39G48DQNQ",
          "message": "<5>Apr 6 16:38:23 root: zhongguoren 4",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:38:23.350"
        },
        "sort": [ - 
          1680770303350,
          "644eb870-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "a562e070-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 45404,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVJC1YT9SPVWG81CJGBMXG",
          "message": "<5>Apr 6 16:40:12 root: zhongguoren 5",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:40:12.534"
        },
        "sort": [ - 
          1680770412534,
          "a562e070-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "a609f5e0-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 33127,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVJD1Y45HAS4N7ZN2YXWS0",
          "message": "<5>Apr 6 16:40:13 root: zhongguoren 6",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:40:13.629"
        },
        "sort": [ - 
          1680770413629,
          "a609f5e0-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "a6a328a0-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 41301,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVJE1A0ER5S5BZ8DJKW2A6",
          "message": "<5>Apr 6 16:40:14 root: zhongguoren 7",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:40:14.633"
        },
        "sort": [ - 
          1680770414633,
          "a6a328a0-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "a73c3451-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 36916,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVJF0PXMNPGKVZ7YEJ1JNY",
          "message": "<5>Apr 6 16:40:15 root: zhongguoren 8",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:40:15.637"
        },
        "sort": [ - 
          1680770415637,
          "a73c3451-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "a7dfc750-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 140,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 42778,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVJG26BRZG09JSSA96S329",
          "message": "<5>Apr 6 16:40:16 root: zhongguoren 9",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:40:16.708"
        },
        "sort": [ - 
          1680770416708,
          "a7dfc750-d456-11ed-aa52-000c29f6b211"
        ]
      },
      { - 
        "_index": "12_149",
        "_type": "_doc",
        "_id": "a894bf71-d456-11ed-aa52-000c29f6b211",
        "_score": null,
        "_source": { - 
          "elap_accounted_message_size": 141,
          "elap_source_input": "636c90d3246ca975ec999b24",
          "streams": [ - 
            "6969696969697379736c6f67",
            "6969696969727379736c6f67"
          ],
          "elap_remote_port": 53090,
          "elap_source_node": "5bb5d963-7fdd-4e2d-bfdf-a5069bb07097",
          "source": "192.168.113.195",
          "elap_message_id": "01GXAVJH78H4GCQ44KNZQ1MQ0H",
          "message": "<5>Apr 6 16:40:17 root: zhongguoren 10",
          "elap_remote_ip": "192.168.113.195",
          "timestamp": "2023-04-06 08:40:17.895"
        },
        "sort": [ - 
          1680770417895,
          "a894bf71-d456-11ed-aa52-000c29f6b211"
        ]
      }
    ]
  }
}

4.Search after二次查询

可以根据sort值带入到后续得search after中，以做到从某点进行再次查询。

GET index_1/_search
{
  "size": 10,
  "query": {
    "match": {
      "message": "zhongguoren"
    }
  },
  "search_after": [
    1680770417895,
    "a894bf71-d456-11ed-aa52-000c29f6b211"
  ],
  "sort": [
    {
      "timestamp": "asc"
    },
    {
      "_id": "asc"
    }
  ]
}

返回结果中，得到11-20条数据。

总结

我也尝试过，直接从16条的sort字段值进行查询，同样可以查询到17-20的值，所以，我们可以进行改造，默认仍然是10000条的max_result_window，兼顾小批量数据的查询和搜索，当查询从66页往后时，每次翻页或者跳页我们要能获取到超过10000的最后一条信息的id和时间戳，由此解决es深度查询翻页的问题。

明天找产品聊聊实现逻辑。