Creating a New Model
To begin leveraging the power of Marketing Mix Modeling, you first need to create a new model. The process is a guided, step-by-step configuration that ensures your model is tailored to your specific business needs.
1. Model Name
Give your model a descriptive name. This helps you easily identify it in the future, especially if you plan to create multiple models for comparison.
2. Category Selection
Next, you'll choose the categorization level for your model from your custom categories. This allows the model to be built around specific marketing tactics or objectives.
2a. Subcategory Configuration
Excluding: affiliate subcategories should be excluded from the model because it is a derivative of what the model is trying to predict. In data science terms, this would be introducing leakage into the model.
Subcategory Prior: if you have run an incrementality experiment or have knowledge about what you think the approximate efficiency of the subcategory is, then you can input it as a prior to the model to guide its learning to align with your sense of its performance.
3. KPI Selection
Choose the primary business metric, or Key Performance Indicator (KPI), that you want to model. You can select from various metrics, including Revenue, Orders, and Profit, with options for both total and new customer data (NC Revenue, NC Orders, NC Profit). By default, NC Revenue is selected because generally that is most strongly correlated with marketing spend.
4. Time Granularity
Select the time period you want your model to analyze. You can choose from Daily, Weekly, or Monthly granularity, which dictates the level of detail and responsiveness of your model. By default, weekly is selected to enable
5. Model Start Date
Select the historical date from which the model should begin its analysis. This is the starting point for the time-series data used to train the model.
6. Include External Factors
You can enhance your model's accuracy by including additional business factors. These include data from platforms like Amazon, as well as internal factors such as Discount Percent and Email/SMS campaigns, which are crucial drivers of KPI.
7. Adding Special Historical Events
Finally, you have the option to add special historical events to your model. This helps the model account for significant one-time occurrences, such as a Product Launch or a major marketing initiative, ensuring these events are properly factored into the analysis.
Monitoring and Managing Your Models
Once you create a model, it will appear in the Models page as a new row with its specific configuration. The Status of the model will be updated automatically to show its current state.
Requested: The model configuration has been saved and is waiting to be processed.
Triggered: The model has started the training process.
Completed: The model has finished training successfully and the results are ready to be viewed.
Failed: The model training process encountered an error.
After a model successfully completes its training, the View Runs button will be enabled under the Actions column. Clicking this button allows you to access the detailed results and insights for that specific model run.
If you need to remove a model, you can do so by clicking the Delete icon in the Actions column. This will permanently remove the model and its associated data.
Analyzing a Specific Model Run
Each time a model is run, it goes through a validation process to ensure accuracy. By default, the model uses the last three time periods as a test period to validate its predictions against actual historical data. You can analyze any historical run up to its last training point, but we recommend using the most recently updated model for the most accurate insights.
On the overview of model run pages, you will find a confidence score for both the test and training periods. This score combines multiple metrics into a single, easy-to-understand value. It provides a quick way to gauge the overall quality and reliability of each specific model run.
Model Performance Metrics
Each model run provides a set of key metrics to help you assess its performance. These metrics are calculated for both the training period (the historical data used to build the model) and the test period (the most recent data used for validation).
CRPS (Continuous Ranked Probability Score): Measures the overall accuracy of the probabilistic forecast. A lower score indicates a more accurate prediction.
MAPE (Mean Absolute Percentage Error): Indicates the average percentage difference between the predicted and actual values. A lower MAPE signifies a more accurate model.
Pearson Correlation: Measures the linear relationship between the predicted and actual values. A value closer to 1.0 indicates a stronger positive correlation and a better fit.
R-squared (R²): Represents the proportion of variance in the dependent variable that is predictable from the independent variables. A higher R² value indicates that the model's predictions align more closely with the actual data.
Historical Fitting Plots
The platform provides a visual comparison of the model's performance. The Historical Fitting Plot shows the Predicted KPI values against the Actual KPI values over time. This visualization helps you understand how well the model's predictions align with real-world results. The shaded area around the predicted values represents the confidence interval, which shows the range of potential outcomes and the certainty of the prediction.
Analyze Your Reallocations
Each completed model run comes with a default optimized reallocation recommendation. To view these insights, click the View Reallocations button under the Actions column.
Understanding the Reallocation Overview
The reallocation page starts with a high-level overview of the entire budget reallocation. This includes:
Reallocation Name: The name you gave to the reallocation.
Type: Indicates if the reallocation is a Simulation or an Optimization.
Total Budget Allocation: The total amount of budget that was allocated in the reallocation scenario.
Total KPI Impact: The projected total KPI value that will be generated after applying the reallocation.
Understanding Reallocation Breakdown
The Reallocation view provides a detailed breakdown of how each subcategory's budget should be adjusted from its last historical spend. This section presents a clear, actionable plan for optimizing your marketing spend for the next period. For example, the recommendation might suggest moving a media subcategory from $33K to $42K to improve performance. Each subcategory can be expanded to see the reallocated budget for its specific campaigns.
Key Reallocation Metrics
Budget at Model Run: This metric shows the last budget amount for each subcategory from the model's training period. It serves as the baseline for the reallocation recommendations.
Reallocated Budget: This is the new budget amount that the model recommends for the next period.
Marginal Lift: This is the most crucial metric in the reallocation view. It represents the expected incremental return you would get from the very next dollar invested in a specific subcategory. The higher the marginal lift, the more probable it is that the model will recommend increasing the budget for that subcategory in the upcoming period.
Expected KPI: This shows the projected KPI if you were to continue with the same budget as the last period.
Reallocated KPI: This shows the projected KPI if you apply the recommended reallocation strategy.
By analyzing these metrics, you can confidently evaluate the model's suggestions and make data-driven decisions to optimize your budget for maximum return.
Visualizing Your Reallocations
For a deeper analysis, each subcategory offers a curve plot that visualizes the relationship between budget and KPI over time. This plot helps you understand the marginal impact of spend and the point of diminishing returns.
The plot includes several key elements:
Historical Data: A scatter plot shows each historical spend and KPI data point. This allows you to see the real-world performance of your previous budgets.
Fitted Curve: The blue line represents the model's fitted curve, showing the optimal relationship between budget and KPI. It's a key visualization for understanding the concept of diminishing returns.
Reallocation Recommendation: The model's recommended reallocation change is plotted as two squares: the starting point (Budget at Model Run) and the recommended ending point (Reallocated Budget). This makes it easy to see the suggested adjustment and its projected impact on the curve.
Creating New Reallocations: Optimization vs. Simulation
In addition to the default reallocation recommendation, you have the flexibility to create your own budget scenarios. Clicking the Create New Reallocation button will give you two primary options: Simulation and Optimization.
1. Simulation: The "What If" Scenario
The Simulation option lets you answer questions like, "What if I change my Meta Prospecting budget by $10K?" This allows you to manually adjust the budget for any subcategory and immediately see the projected impact on your KPI. It's a powerful way to test specific hypotheses and understand the potential outcomes of your planned changes.
Keep in mind that the simulated budget values should be within the historical range of the chosen category to ensure the model's predictions remain accurate.
2. Optimization: Finding the Best Allocation
The Optimization option answers the question, "What is my optimal spend allocation if I use a total budget of $100K?" This method automatically distributes your budget across all subcategories to achieve the best possible KPI, based on the model's insights.
This feature allows you to set specific constraints for each subcategory, giving you granular control over the optimization process.
Spend to Reallocate: This is the starting budget point for each subcategory. The model will begin its optimization from this value.
Max Decrease %: This constraint sets a limit on how much the model can decrease the budget for a subcategory.
Max Increase %: This constraint sets a limit on how much the model can increase the budget for a subcategory.
Lock: You can lock a subcategory to prevent the optimization from making any changes to its budget. This is useful for fixed budgets or strategic categories.
Once you've created a new reallocation, whether through simulation or optimization, it will appear on the reallocation page for that specific model within a few minutes. You can then analyze it in the same way as the default recommendation.