Computer Vision for the Built Environment

The course is an introduction to Visual Machine Perception technology — and specifically Computer Vision and Machine Learning (CV-ML) — for the built environment. It explores fundamentals and latest trends in this technology both in research and products, in tight reference to design, construction, and operation and management. It considers the current and potential impact of this technology on achieving sustainability goals, such as those related to reuse, circularity, and performance-based lifecycle, as well as the organizational considerations behind development and adoption.

About

In recent years, a lot of discussion has been sparked in AEC (Architecture, Engineering, and Construction) on CV-ML for the built environment. Despite advancements in this interdisciplinary field, fundamental questions remain about adopting and adapting CV-ML technology — this course equips students with the rudimentary knowledge of how this technology works and the essential points to consider when applying it to this specific domain.

The growing availability of sensors that collect visual data on commodity hardware, combined with the off-the-shelf, generalization, and zero-shot capabilities of large multimodal models, is creating pressure to identify how new technology can increase efficiency and decrease risk in this trillion-dollar industry. Cautious, well-thought steps are needed for such technologies to thrive in an industry that has historically shown inertia toward technological adoption, while still driving sustainability goals.

The course unfolds in two interwoven storylines:

• The first introduces fundamentals in computer vision and machine learning technology as building blocks for developing related applications, discussed with respect to the latest developments (e.g., diffusion and large language neural models) and their impact on the final solution.
• The second consists of AEC processes — architectural design, construction, and operation and management — which drive the application of the technological storyline.

Students will see the importance of accounting for application requirements when designing a CV-ML system, as well as their impact on the underlying building blocks. Guest speakers from both the CV-ML and AEC domains complement the lectures.

Learning goals

By the end of the course, students will develop computational thinking related to visual machine perception applications for the built environment and the AEC domain. Specifically, they will:

• Gain a fundamental understanding of how this technology works and its impact on AEC and the built environment through example applications.
• Be able to identify limitations, pitfalls, and bottlenecks in these applications.
• Critically think through solutions for the above issues.
• Acquire hands-on experience creatively designing an application given a base system, through in-class demos and at-home assignments.
• Use this course as a stepping-stone or entry point to CV-ML–intensive courses offered in CEE and CS.

Prerequisites / notice

The course does not require any background in CV-ML, computer science, coding, or the AEC domain. It is designed for students of any background, while still engaging advanced students in these topics. Students will engage with and even implement their own code, tightly supported by tutorials and assignments.

Performance evaluation

Grading combines tutorials, assignments, and a final project. Tutorials engage students more deeply in algorithmic and application aspects and are evaluated on understanding of the presented material. Assignments require critical thinking, research into prior work, and/or hands-on interaction with a pre-existing system or codebase. The final project asks students to creatively design and develop an application based on course material; there is no final exam. Students may be evaluated with a letter grade or credit/no credit.

Grading weights: 42% assignments (6% each), 6% tutorials (1% each), and 52% final project (10% proposal, 12% midterm, 30% final submission). Course project reviews are assigned to a team member other than the student's TA to reduce bias, with feedback from project supervisors and the assigned TA feeding into a final grading discussion. All members of a team receive the same grade unless there is clear evidence of unequal participation.

Late submission policy: students have a total of 3 late days across the course. Each day used reduces that deliverable's grade by 15%, compounding for consecutive days (e.g., two late days: grade × 0.85 × 0.85). In case of genuine emergencies, contact the teaching team — some documentation may be requested. Late project reports incur the same penalty for all team members. No late submissions are accepted for project presentations.

Over the quarter, students work on a project related to a topic in Computer Vision for the Built Environment in collaboration with a supervisor, in groups of 2–4. A list of project ideas is provided, and students are expected to implement and test algorithms, covering aspects such as dataset collection, annotation, user testing, and evaluation metrics. Full details are on Canvas.