# Enhancing Probabilistic Time Series Forecasting with Conformal Prediction: An Empirical Study on Uncertainty Quantification
## Motivation
In recent times of global turmoil, time series forecasting has become an increasingly important and relevant area of study. While forecasting point estimates remain fundamental, advances in computing, alongside significant events such as COVID-19, the Russia-Ukraine war, inflation and the Israeli crisis, have intensified the focus on probabilistic forecasting (Makridakis et al., 2018; Makridakis et al., 2020; Makridakis et al., 2022a). For example, in financial risk management, precise uncertainty estimates are crucial for effective portfolio management and for quantifying risks. Inaccurate prediction intervals can lead to misallocation of resources and increased risk exposure. Similarly, in healthcare and disaster management, accurate uncertainty quantification can improve resource allocation and crisis management (Makridakis & Bakas, 2016; Raftery, 2016).
Classical forecasting methods (Box & Jenkins, 1976) often face challenges in accurately estimating prediction intervals (Makridakis et al., 2018; Grushka-Cockayne & Jose, 2020) and typically rely on restrictive distributional assumptions. With the rise of Machine Learning (ML) in forecasting (Makridakis et al., 2022), which frequently involves black-box models, quantifying uncertainty has become increasingly complex, calling for a need to develop methods that address model misspecification while maintaining computational efficiency, especially for large and intricate ones (Martin et al., 2024).
Recent advancements in Machine Learning, particularly Conformal Prediction (CP), offer a distribution-free approach with guaranteed coverage to quantifying uncertainty. Introduced by Gammerman, Vovk and Vapnik (1998) and further extended to the time series domain, CP demonstrates promising capabilities for addressing the challenges faced in Probabilistic Forecasting.
Therefore, enhancing Probabilistic Time Series Forecasting methods with Conformal Prediction offers a novel opportunity to achieve more reliable prediction intervals, which is crucial in high-risk scenarios and in an era where Machine Learning is rapidly influencing the world.
## Aim and Objectives
### Aim
The aim of this project is to provide an in-depth literature review on Conformal Prediction (CP) in the context of uncertainty quantification in time series, and to offer additional empirical evidence on its effectiveness. This will be achieved by applying some of the latest CP methods to multiple datasets and benchmark models that attempt to replicate the conditions of the latest M5 uncertainty competition (Makridakis et al., 2022).
### Objectives
The objectives of the project are the following:
***Review and analyze Conformal Prediction methods in the context of time series**: conduct a thorough literature review of current CP techniques in the context of time series.
***Empirical test**: implement `ACI` and `EnbPI` on 2-3 time series forecasting models using 2-3 diverse datasets from the Monash Time Series Forecasting Archive that includes datasets and models used in the M5 uncertainty competition.
***Evaluate and compare performance**: compare the prediction intervals generated by the CP-enhanced models with those from 2-3 benchmark methods. Use relevant evaluation metrics such as those used in to assess the accuracy and reliability of these prediction intervals.
***Document and present findings**: summarise the results, highlighting the empirical performance of CP methods compared to benchmarks. Provide insights into the practical benefits and limitations of CP for uncertainty quantification in time series forecasting.
## Stakeholders
The proposed project aims to benefit a wide range of stakeholders in both academic and practical domains.
In academia, researchers might benefit from a thorough literature review alongside the additional empirical evidence provided by the project. This could contribute to the fields of Machine Learning and Forecasting by strengthening previous findings and encouraging further research on Conformal Prediction.
In academia, researchers might benefit from a thorough literature review alongside the additional empirical evidence provided by the project. This could contribute to the fields of Machine Learning and Forecasting by strengthening previous findings and encouraging further research on Conformal Prediction, specifically in the time series domain.
In industry, stakeholders across various sectors might gain from additional empirical tests on using Conformal Prediction to estimate more reliable prediction intervals. The existing literature on Conformal Prediction in the context of time series is still nascent and more empirical tests are needed to faciliate further adoptions.
In industry, stakeholders across various sectors might gain from additional empirical tests on using Conformal Prediction to estimate more reliable prediction intervals. The existing literature on Conformal Prediction in the context of time series is still nascent and more empirical tests are needed to faciliate further adoption.
Lastly, academic advisors and data providers are crucial stakeholders for the success of this project. Advisors will offer guidance on the project development and Data providers will facilitate access to the datasets necessary for testing and validating the proposed methods.
In summary, the project’s stakeholders include businesses, academic researchers, data providers and academic advisors.
## Related Work
Conformal Prediction (CP) or Conformal Inference (CI) is a framework that leverages past data to predict confidence levels for new instances (Shafer et al., 2008). This framework addresses several challenges associated with traditional statistical methods and Machine Learning techniques such as sensitivity to assumptions, trade-offs between coverage and interval width and computational complexity (Manokhin, 2023). Despite its advantages, CP's reliance on the assumption of exchangeability poses challenges in time series applications, where observations are often dependent and distribution shifts occur.
Initial efforts to extend the CP framework to time series data focused on minimizing dependency assumptions that limit the exchangeability requirement. Balasubramanian et al. (2014) proposed addressing this by assuming that each observation depends only on those within a given lag $T \in \mathbb{N}$ or window, rather than all past observations. This modified CP framework has been adopted in several studies. For instance, Kath and Ziel (2021) applied it to short-term electricity price forecasting, finding it more reliable than methods such as quantile regression averaging and empirical error distribution approaches. Similarly, Wisniewski, Lindsay, and Lindsay (2020) used it for financial time series, showing that CP yields comparable or slightly better results than quantile regression in forecasting net positions. Additionally, Kowalczewski (2019) enhanced this approach by incorporating normalization in the context of energy forecasting.
More recent work has broadened CP to address a variety of distribution shifts and dependency structures, including those found in time series and other complex data settings trying to move beyond exchangability (Tibshirani et al., 2019; Podkopaev & Ramdas, 2021; Barber et al., 2022).
In particular, several different approaches have emerged that don't rely on exchangability.
Xu and Xie (2021) developed Ensemble batch Prediction Intervals (`EnbPI`), a method that updates residuals of ensemble predictors during prediction to more accurately calibrate prediction intervals without splitting data allowing `EnbPI` to maintain desired $1 - \alpha$ coverage for different types of time series. `EnbPI` rapidly increased popularity and is currently implemented in the open-source libraries Amazon Fortuna (Detommaso et al., 2023) and MAPIE (Taquet et al., 2022). Moreover, Building on top of Barber et al. (2022), due to limitations of `EnbPI`, Xu and Xie (2023) introduced Sequential Predictive Conformal Inference ( `SPCI`) which utilizes the feedback structure of prediction residuals in the sequential prediction problem to obtain desired coverage.
On the other hand, Adaptive Conformal Inference (`ACI`), proposed by Gibbs and Candès (2021) and its subsequent variants (Gibbs and Candès, 2022; Zaffran et al., 2022; Feldman et al., 2023; Angelopoulos et al., 2023) offer a significant advancement by providing approximately calibrated prediction intervals by adjusting a time-varying nominal miscoverage rate or employs an online gradient descent algorithm for calibration. Additionally, Bellman Conformal Inference (`BCI`), introduced by Yang et al. (2024), addresses the issue of `ACI` of not explicitly optimize average interval lengths by adjusting parameters to balance interval length and coverage rates, potentially offering more efficient intervals compared to `ACI` when nominal intervals are poorly calibrated.
### Gaps and Shortcomings
A notable shortcoming in existing Conformal Prediction (CP) methods is the limited empirical validation across diverse scenarios. While theoretical advancements are significant, there is a need for practical evaluations to confirm effectiveness. However, these advancements lack comprehensive empirical testing across different forecasting models and datasets in real-world situations, which is essential for broader adoption. Additionally, there is a gap in the literature regarding a thorough review of the latest developments in CP, particularly in the context of time series forecasting. Although the proposed models have their own limitations, which researchers are actively addressing, this project focuses on addressing the missing aspects—conducting an in-depth literature review and empirical demonstration.
## Data
To achieve the project's aim of enhancing uncertainty quantification with Conformal Prediction (CP) in time series forecasting, datasets will be sourced from the Monash Time Series Forecasting Archive (Godahewa et al., 2021). Curated by leading expert Rob J. Hyndman and the Monash University team, this archive offers a meticulously curated collection of time series datasets essential for benchmarking and evaluating forecasting models. Monash University’s involvement in the M-competitions further underscores the archive's authority and relevance.
### The Monash Time Series Forecasting Archive
The Monash Time Series Forecasting Archive comprises 30 distinct datasets, with 58 variations based on different time frequencies and the handling of missing values. This diversity will provide a comprehensive testing ground across various real-world scenarios, including finance, energy and healthcare. The archive’s meticulous curation will ensure high-quality data, and its GitHub repository will offer detailed instructions for integrating and evaluating new forecasting models against baseline metrics. This approach will guarantee data quality and model comparability, making the archive an ideal resource for the project’s empirical analysis.
## Methods
To achieve the aims of this project, a quantitative study with two Conformal Prediction (CP) methods will be employed. The selected methods are:
* Ensemble Bootstrap Prediction Interval (`EnbPI`): Developed by Xu & Xie (2021), EnbPI addresses the challenge of non-exchangeable data in time series by adding a sequential aspect to the CP framework. This method is designed for robust uncertainty quantification in time series forecasting and is accessible through the open-source MAPIE library, making it an ideal candidate for empirical testing. EnbPI's design ensures that prediction intervals remain valid even in the presence of dependent data.
* Adaptive Conformal Inference (`ACI`): Proposed by Gibbs and Candès (2021), ACI dynamically adjusts prediction intervals to accommodate distribution shifts within time series data. Its ability to adapt to shifts in the data distribution without sacrificing the validity of prediction intervals makes it particularly suited for time series with general dependency structures. The method’s flexibility is further enhanced by its extension, AgACI, which eliminates the need for tuning hyperparameters by using online expert aggregation. ACI has been shown to improve the efficiency of prediction intervals in time series settings, as demonstrated in synthetic experiments and real-world applications like electricity price forecasting.
These methods were previously compared in a study by Zaffran et al. (2022), which serves as an inspiration for this project. The study highlighted the strengths and limitations of each method, underscoring their potential for further empirical investigation in the context of time series forecasting. Moreover, due to their current traction, it makes sense to use these methods rather than others proposed in the literature.
In addition to these CP-based methods, traditional methods commonly used in probabilistic forecasting will be employed for comparison. These conventional approaches will provide a baseline against which the performance of CP methods can be measured, ensuring a comprehensive evaluation of their effectiveness in uncertainty quantification.
The availability of code for all two CP methods ensures they can be rigorously tested against these traditional benchmark models. Each method represents a different approach to uncertainty quantification in time series forecasting, offering valuable insights into their applicability and effectiveness across different scenarios compared to existing probabilistic methods.
The availability of code for all two CP methods ensures they can be rigorously tested against these traditional benchmark models. Each method represents a different approach to uncertainty quantification in time series forecasting, offering valuable insights into their applicability and effectiveness across different scenarios compared to existing methods.
## Work plan
The proposed project will be completed over two months and is divided into two main phases: Phase 1 and Phase 2, with the work in Phase 1 informing Phase 2.
### Phase 1 - Month 1
#### Introduction Writing
* Key Step: Drafting the introduction to outline objectives, background, and significance.
* Duration: this step is estimated to take few days, depending on how detailed the introduction needs to be. Further
* Key Step: Drafting the introduction to outline objectives, background and significance.
* Duration: this step is estimated to take few days, depending on how detailed the introduction needs to be. Further revisions are required to include a quick summary of key findings.
#### Literature review
* Key step: Conducting a comprehensive review of existing research to understand the current state of knowledge.
* Duration: this step is estimated to take 1-2 weeks, depending on the volume of literature and depth of review required.
#### Data Collection and Assessment
Key Step: Gathering relevant data and assessing its suitability for the experimental setup.
Duration: given that Monash Time Series Archive does include the few days, depending on the availability and accessibility of the data.
*Key Step: Gathering relevant data from the Monash Archive and assessing its suitability for the experimental setup.
*Duration: given that the Monash Time Series Archive does include the few days, depending on the availability and accessibility of the data.
#### Methodology Development
Key Step: Developing the research methodology, including techniques and processes.
Duration: 1 week, as it involves planning and detailing the approach for the experimental phase.
*Key Step: Developing the research methodology, including techniques and processes.
*Duration: 1 week, as it involves planning and detailing the approach for the experimental phase.
#### Validation
Key Step: Validating research questions, methodology, and data assessment to ensure readiness for Phase 2.
Duration: few days, to ensure that the methodology and data are robust and feasible.
*Key Step: Validating research questions, methodology, and data assessment to ensure readiness for Phase 2.
*Duration: few days, to ensure that the methodology and data are robust and feasible.
### Phase 2 - Month 2
#### Data Processing
* Key Step: Cleaning and preparing data for analysis.
* Duration: few hours.
#### Model Building and Test
Key Step: Developing and testing models based on the validated methodology.
Duration: 1-2 weeks, as it involves iterative testing and refinement of models.
*Key Step: Developing and testing models based on the validated methodology.
*Duration: 1-2 weeks, as it involves iterative testing and refinement of models.
#### Analysis
Key Step: Analyzing results and developing conclusions.
Duration: 1 week, as it involves interpreting results and synthesizing findings.
*Key Step: Analyzing results and developing conclusions.
*Duration: 1 week, as it involves interpreting results and synthesizing findings.
#### Conclusion Writing
* Key Step: Drafting the conclusion to summarize findings, discuss implications, and suggest future research.
* Duration: A few days, as it summarizes the project’s outcomes and implications.
#### Key Milestones for Evaluation
End of Week 1 (Phase 1): Completion of Introduction Writing and start of Literature Review.
Milestone: Draft of the introduction should be complete.
End of Week 2 (Phase 1): Completion of Literature Review and Gap Analysis.
Milestone: Identification of research gaps and completion of the literature review.
End of Week 3 (Phase 1): Completion of Data Collection and Assessment, and Methodology Development.
Milestone: Data should be collected and assessed, and methodology should be developed.
End of Week 4 (Phase 1): Completion of Validation.
Milestone: Validation of the research questions, methodology, and data assessment.
End of Week 5 (Phase 2): Completion of Data Processing.
Milestone: Data should be cleaned and ready for analysis.
End of Week 6 (Phase 2): Completion of Model Building and Testing.
Milestone: Initial models should be built and tested.
End of Week 7 (Phase 2): Completion of Analysis.
Milestone: Results should be analyzed, and initial findings should be developed.
End of Week 8 (Phase 2): Completion of Conclusion Writing and overall project wrap-up.
Milestone: Conclusion should be drafted, and the final project report should be prepared
**End of Week 1 (Phase 1):**
***Milestone:** Draft the Introduction.
* Task: Draft of the introduction and begin the literature review.
**End of Week 2 (Phase 1):**
***Milestone:** Completion of Literature review.
* Task: Identify research gaps and complete the literature review.
**End of Week 3 (Phase 1):**
***Milestone:** Completion of Data Collection and Methodology Development.
* Task: Collect and assess data, and develop the methodology.
**End of Week 4 (Phase 1):**
***Milestone:** Validation of Research Framework.
* Task: Validate the research questions, methodology, and data assessment.
**End of Week 5 (Phase 2):**
***Milestone:** Completion of Data Processing.
* Task: Clean the data and prepare it for analysis.
**End of Week 6 (Phase 2):**
***Milestone:** Completion of model building and testing.
* Task: Build and test initial models.
**End of Week 7 (Phase 2):**
***Milestone:** Completion of Analysis.
* Task: Analyze results and develop initial findings.
**End of Week 8 (Phase 2):**
***Milestone:** Completion of Conclusion Writing and Project Wrap-Up.
* Task: Draft the conclusion, complete the final editing, revisions, proofreading, and finalize the project report.
## Risk assessment
Effective risk management is crucial for the success of the proposed project. The risks identified are categorized into two types: project-based risks and non-project-based risks.
Effective risk management is crucial for the success of the proposed project. Three project-based risks have been identified.
### Project-Based Risks
#### Availability of Resources
### Availability of Resources
* Risk: Types of might be unavailable or incomplete, and necessary hardware might not meet technical requirements.
* Mitigation: Validation of data acquisition sources was conducted to ensure reliability, thus the choice of Monash Time Series Archive. Hardware requirements have been pre-assessed ensuring they can run on the hardware currently in use which is Mac M1 (2020).
#### Scope Creep
### Scope Creep
* Risk: The scope of the project could lead to an unmanageable workload such as extensive data preprocessing.
* Mitigation: The project’s aims and objectives will be adjusted to a nicher case if needed without compromising the core research question. This approach maintains focus while ensuring the project remains feasible.
#### Feasibility and Complexity
### Feasibility and Complexity
* Risk: The complexity of the selected models may exceed current skills, leading to delays.
* Mitigation: Iterative project planning will allow the focus to be narrowed if necessary. For example, focusing on a specific industry could reduce the number of datasets and simplify the models.
### Non-Project-Based Risks
## Expected results
#### Personal Emergencies
This project leverages approaches from the M5 Uncertainty competition, particularly in quantifying uncertainty in time series forecasting through Conformal Prediction. The focus will be on generating 9 nominal probability levels
$u \in \{0.005, 0.025, 0.165, 0.250, 0.500, 0.750, 0.835, 0.975, 0.995\}$ to predict median values and construct 4 central prediction intervals (PIs) at confidence levels of $50\%$, $67\%$, $95\%$, and $99\%$. These outputs are expected to help accurately characterize the distribution's center and tails, offering a comprehensive understanding of forecast uncertainty (Makridakis et al., 2022).
* Risk: Unexpected personal events, such as the illness or death of a close relative (grandmother), could disrupt the project timeline.
* Mitigation: A buffer period has been built into the work plan to accommodate potential delays. Early project initiation and strict adherence to milestones will help maintain progress.
However, it is important to approach the expected results with caution. Time series data is inherently complex, with varying time dependencies, different forecasting horizons, and industry-specific factors, making it challenging to generalize findings. The outcomes of this project should be taken with a grain of salt, recognizing that they may be valid within the specific context of the data used but may not be directly applicable to other datasets or domains without careful consideration.
#### Health and Performance Issues
* Risk: Decreased performance due to health-related issues could impact the quality and timeliness of the work.
* Mitigation: Preventative measures, including a balanced diet and regular physical exercise, will be prioritized to maintain health and performance throughout the project.
This does not mean that the project is without value. On the contrary, it serves as a foundation for further exploration and research in the field of Uncertainity Quantification in time series setting. By aligning with key future directions from the M5 competition—such as further developing Machine Learning methods for forecasting and enhancing reproducibility and practical implementation—the project aims to incentivize additional work in this area. Moreover, it seeks to raise awareness among academics and practitioners about the importance of understanding and communicating the full extent of uncertainty in forecasting, particularly in light of unpredictable events like COVID-19, which can lead to significant and unforeseen risks.
## Evaluation
## Expected result
The evaluation of this project focuses on the precision of probabilistic forecasts, inspired by the M5 Uncertainty competition. The primary metric used is the Scaled Pinball Loss (SPL), calculated for each time series and quantile. SPL assesses forecast accuracy by penalizing deviations from actual values, using the following formula:
where $Y_t$ is the actual value at time $t$, $Q_t(u)$ is the forecasted quantile, $h$ is the forecasting horizon (which can be 28 days, as suggested by the competition, or adjusted based on the datasets), $n$ is the length of the training sample, and $\mathbf{1}$ is an indicator function. The denominator of SPL is scaled by the average absolute change in actual values during the training period, ensuring comparability across series with different scales and datasets, making it neutral (Hyndman & Koehler, 2006).
The Weighted Scaled Pinball Loss (WSPL) will then aggregate SPL across all the time series analyzed and the nine quantiles to rank the performance of the methods. WSPL is computed as:
where $\text{n_series}$ is the number of time series analyzed, and each series is weighted equally. This approach differs from the WSPL proposed in the M5 competition, which weights each series based on recent actual sales. While equal weighting might be further refined in future work, it provides an effective starting point.
To evaluate if the aims of the project are achieved, the `ACI` and `EnbPI` methods will be benchmarked, and their WSPL scores compared against traditional methods such as Naive, Seasonal Naive (sNaive), Simple Exponential Smoothing (SES), Exponential Smoothing (ES), AutoRegressive Integrated Moving Average (ARIMA), and Kernel density estimates. Although this choice mirrors the M5 competition and may be subject to debate, the author believes it highlights the advantages of Conformal Prediction, particularly as traditional benchmarks, which are widely used, often assume normally distributed forecast errors, an assumption rarely met in practice.
1. Makridakis, S., Spiliotis, E., Assimakopoulos, V., 2018. The M4 Competi- tion: Results, findings, conclusion and way forward. International Journal of Forecasting 34, 802–808.
2. Makridakis, S., C. Fry, F. Petropoulos, and E. Spiliotis. 2022a. “The Future of Forecasting
Competitions: Design Attributes and Principles.” INFORMS Journal on Data Science 1 (1):
96–113.
3. Makridakis, S., R. J. Hyndman, and F. Petropoulos. 2020. “Forecasting in Social Settings: The
State of the Art.” International Journal of Forecasting 36 (1):15–28.
4. Raftery AE. Use and Communication of Probabilistic Forecasts. Stat Anal Data Min. 2016 Dec;9(6):397-410. doi: 10.1002/sam.11302. Epub 2016 Feb 23. PMID: 28446941; PMCID: PMC5403155.
5. Makridakis, Spyros and Bakas, Nikolas. ‘Forecasting and Uncertainty: A Survey’. 1 Jan. 2016 : 37 – 64.
6. George.E.P. Box and Gwilym M. Jenkins. Time Series Analysis: Forecasting and Control. Holden-Day, 1976.
7. Grushka-Cockayne, Y. & Jose, V.R.R., 2020. Combining prediction intervals in the M4 competition. International Journal of Forecasting, 36(1), pp.178-185. Available at: https://doi.org/10.1016/j.ijforecast.2019.04.015.
8. Makridakis, S., Spiliotis, E., Assimakopoulos, V., Chen, Z., Gaba, A., Tsetlin, I., & Winkler, R.L. (2022). The M5 uncertainty competition: Results, findings and conclusions. International Journal of Forecasting, 38(4), 1365-1385. https://doi.org/10.1016/j.ijforecast.2021.10.009.
9. Martin, Gael M. & Frazier, David T. & Maneesoonthorn, Worapree & Loaiza-Maya, Rubén & Huber, Florian & Koop, Gary & Maheu, John & Nibbering, Didier & Panagiotelis, Anastasios, 2024. "Bayesian forecasting in economics and finance: A modern review," International Journal of Forecasting, Elsevier, vol. 40(2), pages 811-839.
10. Gammerman, A., V. Vovk, and V. Vapnik. 1998. Learning by transduction. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence UAI’98, 148–55. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc.
11. Xu, Chen and Yao Xie (2021). Conformal prediction interval for dynamic time-series. Conference version, The Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021.
12. Isaac Gibbs and Emmanuel Cand`es. Adaptive conformal inference under distribution shift. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=6vaActvpcp3.
13. Yang, Z., Candès, E., & Lei, L., 2024. Bellman Conformal Inference: Calibrating Prediction Intervals For Time Series. arXiv [online]. Available at: https://doi.org/10.48550/arXiv.2402.05203
14. Manokhin, V., 2023. Practical Guide to Applied Conformal Prediction in Python: Learn and apply the best uncertainty frameworks to your industry applications. 1st ed. Packt Publishing.
15. Kowalczewski, J. 2019. Normalized conformal prediction for time series data.
16. Shafer, G. and Vovk, V., 2008. A tutorial on conformal prediction. Journal of Machine Learning Research, 9, pp.371-421.
17. Kath, C., and F. Ziel. 2021. Conformal prediction interval estimation and applications to day-ahead and intraday power markets. International Journal of Forecasting 37 (2):777–99. doi:10.1016/j.ijforecast.2020.09.006.
18. Wisniewski, W., D. Lindsay, and S. Lindsay. 2020. Application of conformal prediction interval estimations to mar-
ket makers’ net positions. In Conformal and Probabilistic Prediction and Applications, PMLR, pp. 285–301.
19. V.N. Balasubramanian, S-S. Ho, and V. Vovk. Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications - Chapter 9: Other Adaptations. Morgan Kaufmann Publishers Inc., San Francisco, CA, 1st edition, 2014. ISBN 0123985374.
20. Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Cand`es, and Aaditya Ramdas. Conformal prediction under covariate shift. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ́e-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/ 8fb21ee7a2207526da55a679f0332de2-Paper.pdf.
21. Aleksandr Podkopaev and Aaditya Ramdas. Distribution-free uncertainty quantification for classification under label shift. arXiv preprint, 2021. arXiv:2103.03323.
22. Rina Foygel Barber, Emmanuel J. Cand`es, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeability. arXiv preprint, 2022. arXiv:2202.13415.
23. Detommaso, G., Gasparin, A., Donini, M., Seeger, M., Wilson, A.G. and Archambeau, C., 2023. Fortuna: A library for uncertainty quantification in deep learning. arXiv preprint arXiv:2302.04019
24. Taquet, V., Blot, V., Morzadec, T., Lacombe, L. and Brunel, N., 2022. MAPIE: an open-source library for distribution-free uncertainty quantification.
25. Isaac Gibbs and Emmanuel Cand`es. Adaptive conformal inference under distribution shift. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=6vaActvpcp3.
26. Isaac Gibbs and Emmanuel Cand`es. Conformal inference for online prediction with arbitrary distribution shifts, 2022.
27. Margaux Zaffran, Olivier F ́eron, Yannig Goude, Julie Josse, and Aymeric Dieuleveut. Adaptive conformal predictions for time series. In International Conference on Machine Learning, pages 25834–25866. PMLR, 2022.
28. Anastasios N Angelopoulos, Emmanuel J Candes, and Ryan J Tibshirani. Conformal PID control for time series prediction. arXiv preprint arXiv:2307.16895, 2023.
29. Shai Feldman, Liran Ringel, Stephen Bates, and Yaniv Romano. Achieving risk control in online learning settings, 2023.
30. Tajmouati, S., B. El Wahbi, A. Bedoui, A. Abarda, and M. Dakkoun. 2021. Applying k-nearest neighbors to time
series forecasting: Two new approaches, arXiv e-prints arXiv:2103.14200.
31. Samya Tajmouati, Bouazza EL Wahbi & Mohamed Dakkon (2024) Applying regression conformal prediction with nearest neighbors to time series data, Communications in Statistics - Simulation and Computation, 53:4, 1768-1778, DOI: 10.1080/03610918.2022.2057538 (Tajmouati et al., 2024).
32. Kelly, M., Longjohn, R., & Nottingham, K., The UCI Machine Learning Repository. Available at: https://archive.ics.uci.edu [Accessed 12 Aug. 2024].
33. Godahewa, R., Bergmeir, C., Webb, G.I., Hyndman, R.J., & Montero-Manso, P., 2021. Monash Time Series Forecasting Archive. In Neural Information Processing Systems Track on Datasets and Benchmarks.
34. Hyndman, R.J. & Koehler, A.B., 2006. Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), pp.679–688. Available at: http://dx.doi.org/10.1016/j.ijforecast.2006.03.001.
