Why must idf.py be run in a directory that contains a CMakeLists.txt file?

idf.py is a front end to CMake. The top-level CMakeLists.txt is what marks a folder as an ESP-IDF project; without it there is no project to configure, build or flash.

A minimal ESP-IDF project looks like this:

my_project/
├── CMakeLists.txt        # top level: makes this folder a project
├── sdkconfig             # created by menuconfig / the first build
└── main/
    ├── CMakeLists.txt    # registers the "main" component and its sources
    └── tasks_main.c

The top-level file is three lines: it sets the minimum CMake version, includes ESP-IDF's project logic and names the project. Every component, including main, has its own CMakeLists.txt that lists its source files and dependencies.

When you type idf.py build, it looks for that top-level file in the current directory and passes it to CMake. In the wrong directory, a parent folder or your home directory, there is nothing to build and the command fails. The rule is simply: cd into the project first.

Go deeper:

• ESP-IDF — Build System: example project — the project layout and the minimal top-level CMakeLists.txt.
• Wikipedia — CMake — what a meta build system does, and how CMakeLists.txt files describe a project.