[mongodb-dev] Query processing code

[Monika Shah]

How can we trace query processing as well as query optimization?

['David Storch' via mongodb-dev]

Hi Monika,

I'm not sure exactly what you mean by tracing query processing, but you
could try enabling verbose logging for the QUERY log component. This will
log lots of detail from the query planner in particular. You can enable
this by connecting to the server with the mongo shell and running the
following:

db.setLogLevel(5, "query");

If you provide some more detail about what you were trying to do, I might
be able to be more helpful!

Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?

[Monika Shah]

Thank you David.
I wanted to ask you that how can I trace source code for query processing.
I want to know how can I contribute for performance metrics. So, I want to
understand path and apply change for testing my idea

Post by 'David Storch' via mongodb-dev
Hi Monika,
I'm not sure exactly what you mean by tracing query processing, but you
could try enabling verbose logging for the QUERY log component. This will
log lots of detail from the query planner in particular. You can enable
this by connecting to the server with the mongo shell and running the
db.setLogLevel(5, "query");
If you provide some more detail about what you were trying to do, I might
be able to be more helpful!
Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?

['David Storch' via mongodb-dev]

Hi Monika,

There are quite a few submodules involved in query processing: the query
planner, the query execution engine (composed of the query exec library and
the aggregation framework libraries), the full text search engine,
geospatial search, the mongos query execution path, and so on. In order for
me to help you find a starting point for reading the source code, could you
give me a little bit more detail about what submodule you are interested or
about the performance metrics you are hoping to add?

Best,
Dave

Post by Monika Shah
Thank you David.
I wanted to ask you that how can I trace source code for query processing.
I want to know how can I contribute for performance metrics. So, I want to
understand path and apply change for testing my idea

Post by 'David Storch' via mongodb-dev
Hi Monika,
I'm not sure exactly what you mean by tracing query processing, but you
could try enabling verbose logging for the QUERY log component. This will
log lots of detail from the query planner in particular. You can enable
this by connecting to the server with the mongo shell and running the
db.setLogLevel(5, "query");
If you provide some more detail about what you were trying to do, I might
be able to be more helpful!
Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?

[Monika Shah]

I appreciate your prompt reply.

I would like to find time, space, data proximity, power for different
components(like per tuple cost, per memory page cost, per IO channel cost)
used by each query plan.

Post by 'David Storch' via mongodb-dev
Hi Monika,
There are quite a few submodules involved in query processing: the query
planner, the query execution engine (composed of the query exec library and
the aggregation framework libraries), the full text search engine,
geospatial search, the mongos query execution path, and so on. In order for
me to help you find a starting point for reading the source code, could you
give me a little bit more detail about what submodule you are interested or
about the performance metrics you are hoping to add?
Best,
Dave

Post by Monika Shah
Thank you David.
I wanted to ask you that how can I trace source code for query processing.
I want to know how can I contribute for performance metrics. So, I want
to understand path and apply change for testing my idea

Post by 'David Storch' via mongodb-dev
Hi Monika,
I'm not sure exactly what you mean by tracing query processing, but you
could try enabling verbose logging for the QUERY log component. This will
log lots of detail from the query planner in particular. You can enable
this by connecting to the server with the mongo shell and running the
db.setLogLevel(5, "query");
If you provide some more detail about what you were trying to do, I
might be able to be more helpful!
Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?

['David Storch' via mongodb-dev]

Hi Monika,

I would recommend reading the code path for the explain command
<https://docs.mongodb.com/manual/reference/command/explain/>. Explain calls
into the Command::explain() virtual method, implemented by CRUD commands
such as find and aggregate. The entry point for explain of a find command,
for example, is here:

https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/commands/find_cmd.cpp#L139

For the aggregate command, the entry point is here:

https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/commands/pipeline_command.cpp#L99

You may also wish to read some of the explain-specific logic which lives in
explain.{h,cpp}:

https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/query/explain.h

Explain currently exposes query-level execution statistics, not
storage-level statistics. Collecting some of the metrics you mentioned may
require some deeper collaboration with the storage layer. However, I hope
that the pointers I've listed above are a good starting point.

Best,
Dave

Post by Monika Shah
I appreciate your prompt reply.
I would like to find time, space, data proximity, power for different
components(like per tuple cost, per memory page cost, per IO channel cost)
used by each query plan.

Post by 'David Storch' via mongodb-dev
Hi Monika,
There are quite a few submodules involved in query processing: the query
planner, the query execution engine (composed of the query exec library and
the aggregation framework libraries), the full text search engine,
geospatial search, the mongos query execution path, and so on. In order for
me to help you find a starting point for reading the source code, could you
give me a little bit more detail about what submodule you are interested or
about the performance metrics you are hoping to add?
Best,
Dave

Post by Monika Shah
Thank you David.
I wanted to ask you that how can I trace source code for query processing.
I want to know how can I contribute for performance metrics. So, I want
to understand path and apply change for testing my idea

Post by 'David Storch' via mongodb-dev
Hi Monika,
I'm not sure exactly what you mean by tracing query processing, but you
could try enabling verbose logging for the QUERY log component. This will
log lots of detail from the query planner in particular. You can enable
this by connecting to the server with the mongo shell and running the
db.setLogLevel(5, "query");
If you provide some more detail about what you were trying to do, I
might be able to be more helpful!
Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?

[Monika Shah]

Link you shared is good to have some basic idea.
Can you share me code of query optimization?

Post by 'David Storch' via mongodb-dev
Hi Monika,
I would recommend reading the code path for the explain command
<https://docs.mongodb.com/manual/reference/command/explain/>. Explain
calls into the Command::explain() virtual method, implemented by CRUD
commands such as find and aggregate. The entry point for explain of a find
https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/commands/find_cmd.cpp#L139
https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/commands/pipeline_command.cpp#L99
You may also wish to read some of the explain-specific logic which lives
https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/query/explain.h
Explain currently exposes query-level execution statistics, not
storage-level statistics. Collecting some of the metrics you mentioned may
require some deeper collaboration with the storage layer. However, I hope
that the pointers I've listed above are a good starting point.
Best,
Dave

Post by Monika Shah
I appreciate your prompt reply.
I would like to find time, space, data proximity, power for different
components(like per tuple cost, per memory page cost, per IO channel cost)
used by each query plan.

Post by 'David Storch' via mongodb-dev
Hi Monika,
There are quite a few submodules involved in query processing: the query
planner, the query execution engine (composed of the query exec library and
the aggregation framework libraries), the full text search engine,
geospatial search, the mongos query execution path, and so on. In order for
me to help you find a starting point for reading the source code, could you
give me a little bit more detail about what submodule you are interested or
about the performance metrics you are hoping to add?
Best,
Dave

Post by Monika Shah
Thank you David.
I wanted to ask you that how can I trace source code for query processing.
I want to know how can I contribute for performance metrics. So, I want
to understand path and apply change for testing my idea

Post by 'David Storch' via mongodb-dev
Hi Monika,
I'm not sure exactly what you mean by tracing query processing, but
you could try enabling verbose logging for the QUERY log component. This
will log lots of detail from the query planner in particular. You can
enable this by connecting to the server with the mongo shell and running
db.setLogLevel(5, "query");
If you provide some more detail about what you were trying to do, I
might be able to be more helpful!
Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?

[Monika Shah]

I have set break point find_cmd.cpp:132 (first line of
find_cmd.cpp::explain() ), and find_cmd.cpp:231.
After that when I debug code, it stop at this line only for
db.collection.findOne() not for db.collection.find( predicate1,
predicate2).count().


Please help me to see how does it verify criteria to prepare plan



















0

Post by 'David Storch' via mongodb-dev
Hi Monika,
I would recommend reading the code path for the explain command
<https://docs.mongodb.com/manual/reference/command/explain/>. Explain
calls into the Command::explain() virtual method, implemented by CRUD
commands such as find and aggregate. The entry point for explain of a find
https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/commands/find_cmd.cpp#L139
https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/commands/pipeline_command.cpp#L99
You may also wish to read some of the explain-specific logic which lives
https://github.com/mongodb/mongo/blob/edc3dfce51b00ba3007f35cf6383efb72bf8b74e/src/mongo/db/query/explain.h
Explain currently exposes query-level execution statistics, not
storage-level statistics. Collecting some of the metrics you mentioned may
require some deeper collaboration with the storage layer. However, I hope
that the pointers I've listed above are a good starting point.
Best,
Dave

Post by Monika Shah
I appreciate your prompt reply.
I would like to find time, space, data proximity, power for different
components(like per tuple cost, per memory page cost, per IO channel cost)
used by each query plan.

Post by 'David Storch' via mongodb-dev
Hi Monika,
There are quite a few submodules involved in query processing: the query
planner, the query execution engine (composed of the query exec library and
the aggregation framework libraries), the full text search engine,
geospatial search, the mongos query execution path, and so on. In order for
me to help you find a starting point for reading the source code, could you
give me a little bit more detail about what submodule you are interested or
about the performance metrics you are hoping to add?
Best,
Dave

Post by Monika Shah
Thank you David.
I wanted to ask you that how can I trace source code for query processing.
I want to know how can I contribute for performance metrics. So, I want
to understand path and apply change for testing my idea

Post by 'David Storch' via mongodb-dev
Hi Monika,
I'm not sure exactly what you mean by tracing query processing, but
you could try enabling verbose logging for the QUERY log component. This
will log lots of detail from the query planner in particular. You can
enable this by connecting to the server with the mongo shell and running
db.setLogLevel(5, "query");
If you provide some more detail about what you were trying to do, I
might be able to be more helpful!
Best,
Dave

Post by Monika Shah
How can we trace query processing as well as query optimization?