Splunk On-Call Erfahrungen

VictorOps

Kommentare: Prior to Bluestem Technologies migration to VictorOps, only a few of our squads were using Pagerduty for on-call incident management. Our major goals and justifications for switching to VictorOps was onboarding our entire technology team onto one on-call solution as well increasing our major incident responsiveness. VictorOps was chosen over PagerDuty for it’s people-centric focus, feature capabilities and cost.

Vorteile:

The transmogrifier has been an extremely useful tool for many of our teams. We've added playbook annotations to most of our alerts which has been a boon for our 24 by 7 support team. The Datadog integration has also shown VictorOps's savviness and people-centric approach to building a product. A Datadog VictorOps incident will send all of the pertinent information and graphs to an on-call users phone. The VictorOps mobile app can also be used as a platform to seamlessly chat with other users. We've also been encouraged by some of VictorOp's new features they have started to roll out. Their Statuspage.io integration has made many in our organization excited about the ability to have a one-stop-shop dashboard to communicate down-time incidents to a diverse group of users within our company. Their brand-new Control Calling feature (a feature that creates a conference call and pages users to join, penned "the new bat signal" by VictorOps) is also an intriguing feature that several of our teams have started to explore. Possibly our favorite aspect of working with VictorOps over PagerDuty is how responsive their team has been to both our support and development requests.

Nachteile:

VictorOps is still lacking some of PagerDuty's clerical and reporting functionality; for example, we no longer have the ability to easily reference a specific incident in the past.

A really good product supported by a great team

Kommentare: VictorOps was the default on-call software for a couple of years for various teams in the organization. It works amazingly most of the time. Only issue that was faced was after Zabbix changed their alerting mechanism which somewhat broke the VictorOps. VictorOps team was quick to devise a solution using webhooks, which worked immediately. They however, took some time to figure out the ack-back solution for Zabbix 4.4, and had a bug which delayed the deployment. However, VictorOps support was really good throughout this time, they stayed in contact, and even let developers do live debugging to figure out the bug, and the solution was solved. All in all, I have nice things to say about VictorOps.

Vorteile:

- Easy to use and setup - Alerting data can be customized (found this feature missing on other competing software) - Ack-back feature (used Zabbix) - Very good mobile app with notifications - Alert snooze option which is a life saver. - Alert routing works perfectly

Nachteile:

- Duplicate alerts happen when alerts flap. - Alerting data contains too much data (relevant information is usually found after scrolling)

Cheap, but that's about it

Kommentare: IMO, Splunk bought VictorOps to tick a checkbox and nothing more. It's cheap, but unless you're totally strapped for cash, look elsewhere.

Vorteile:

VictorOps (now called Splunk On-Call) organizes incidents in a manner similar to a Twitter timeline. It's really convenient being able to browse your history like that. There are a decent number of integrations with third-party services, and the Rule Engine allows you to annotate incoming alerts with additional information such as links to runbooks. In terms of price, you'll pay a lot less than you will for competing products like PagerDuty.

Nachteile:

Since being acquired by Splunk, VictorOps has been left to rot on the vine. There have been virtually no new features. I was willing to excuse that when VictorOps was operating as a smaller company, but not being able to do an in-depth history of incidents in 2021 is just inexcusable. Splunk's high-pressure sales agents will constantly try to upsell you on other Splunk products. SSO integration is still awkward to set up and use, and there still aren't nearly as many integrations as with PagerDuty.

Victory for on-call and alert escalation

Kommentare: For years we have had to manually manage our on call rotations throughout the organization. This linked with the wide range of monitoring tools, problem management has been difficult. VictorOps has allowed us to jump into the future of Outage Management! This has allowed us to reduce our MTTR (mean time to resolution) for outages. I highly recommend this tool!

Vorteile:

VictorOps was easy to deploy, configure teams, and manage on-call rotations. There is a large list of integrations that allow for connections into all our required systems. Included with every integration is a link to their knowledge base. VictorOps is a must have tool for anyone looking to more effectively manage alert escalation and on-call rotations.

Nachteile:

There are only two things i have come across that are lacking. First, setting up overrides for on call management. VictorOps only allows for overrides to be in 30 minutes time blocks. This has caused issues with emergency on call coverage. Second is no having a location, out side the alerts and incidents, to house knowledge base type documents.

VictorOps: Now Brought to You By Splunk

Kommentare: Splunk's purchase of VictorOps gives the opportunity for major improvement and excellent customer support.

Vorteile:

Provides all of the necessary functionality for managing multiple teams of on-call professionals across an organization. App and Slack integration provide a feature-rich experience. override existing schedules, acknowledge or snooze alerts, see detailed history, and run reports.

Nachteile:

Some features are buried in menus, and the calendar is a little hard to use at times. Some options require clicking a specific toggle icon instead of just the text.

The perfect fit

Kommentare: When we went looking for an alerting solution, we wanted something that would make life easier for our folks on-call, as well as something that would help us clean up and make sense of the nigh-impassable flood of alert emails we were seeing constantly from our monitoring. Allowing folks to use their own devices also meant that there was no confusion as to how to reach out to someone, and it's something they always have on them. We were immediately captured by the Twitter-like timeline, so that every person (on-call or not) could view the incoming stream from the entire environment, and the rest of the features common in this space seemed very well-implemented. Since then, they've added tons of new features (the Transmogrifier and Control Calls each were game-changers) that have only made the service better. My director has identified this as the single best product we've implemented in the past two years, based on its impact to our team's efficiency and happiness.

Vorteile:

- Communication-centric design - great for team efforts - Very responsive support and dev team - Easy to set up, easy to use - Lots of integrations available - Contact details in the app makes reaching out to teammates easier in a crisis

Nachteile:

- Some advanced/niche scheduling options are not currently possible - Data on old alerts can be difficult or impossible to obtain, depending on age - Reporting is a work in progress

Awesome Way to manage On Call

Kommentare: VictorOps has been used in our environment for over a year. This has been an invaluable tool to making sure our staff are aware of what's happening in the environment and pulling in alerts from a couple of different systems. The Transmorgrifier is my favorite feature that I cannot find anywhere else. We've completely transformed at least one custom alert so that our techs don't have to dig for relevant information, they can just find it right in front of them. The best non-replicable asset going, however, is the staff and the culture at VictorOps. They genuinely care about their customers and have even worked with us to develop an integration that we wanted to use in our environment.

Vorteile:

Excellent On-call management and escalations, Transmorgrifier, Customer Service

Nachteile:

Potentially, there could be more integrations, but there seem to be more products out there daily that want to send alerts. It is obvious that the dev staff is looking to include more integrations and are extremely responsive to their customer requests even assisting us with working on custom integrations.

VictorOps: Organizing the Chaos of Operations

Kommentare: We recently switched over from PagerDuty to VictorOps while refactoring our monitoring and alerting infrastructure. We wanted to go with a more ChatOps approach and we were ready for something new. The support at VictorOps has been great, and they have responded to every request we've made. They are adding features with every release that are rounding out their offering and adding even more value to their product.

Vorteile:

Lots of integration's. Easy to set up and get started. Great customer service. The Dev and Support teams are very responsive. Transmogrifier!

Nachteile:

Interface can be hard to use. API is missing key features and still rough around the edges.

Outstanding service/app

Kommentare: We've been using VictorOps for close to 2 years now and both the app and the service behind it are fantastic. Being able to funnel all of our alerts into 1 alerting system where we manage schedules is a great feature..but others have this. What really makes VictorOps shine is the timeline feature where you can see full history of alerts in a simple timeline view and then 'loop in' team members for help if needed.

Vorteile:

does what it says on the tin very reliable Easy to understand UI

Nachteile:

Android app looks a little (very little!) dated

Great product, great company

Kommentare: Very customizable, easy to use, and fantastic support. This has really made managing alerts and on-call so much easier, and we've always had helpful and friendly support when needed.

optimally priced on-call software to ensure production fires are immediately acknowledged

Kommentare: We use VictorOps for real time incident reporting. This is integrated using scalyr and splunk and slack. So we get alerts on slack with clickable links from scalyr with the actual incident reports logged into victorops. the service is very reliable, and has never failed us once. I would recommend it without reservations to teams who rely heavily on cloud based systems where real time incident reporting is very important

Vorteile:

works like clockwork, providing a smooth seamless experience. It is incredibly easy to assign on call schedules, and set up alerting. the details is provides about the incidents is very granular and helps provide much needed context. it is fairly inexpensive and integrates really well with several other softwares

Nachteile:

This isn't a con but a good to have feature I should say. basically it is only as good as your configuration and your logging. victor ops on it's own won't detect new incidents based on past incidents. If certain configurations are off, then it would never receive data input from those points, and you would never be notified of potential incidents arising from the same without human intervention.

Add another member to your team with Victor Ops!!

Kommentare: Using VictorOps is like adding another member to your team. By removing the cumbersome problem of over-active alerts, which can cause some serious team fatigue which we experienced prior to using victor ops, you have the ability to add a number of important options relative to the alerts you have created. With the Transmogrifier you can filter through irrelevant alerts with great ease, delivering only the critical alerts to your team. This is of course a process that involves tuning and can take some time. We use Nagios as one of our monitoring tools and the integration with victor ops was seamless. Allowing us to quickly adapt to the ever changing environment that makes up our infrastructure.

Vorteile:

I am really looking forward to our increased use of the Transmogrifier and its ability to attach documentation to alerts on the fly. I can hardly wait until the next alert that rolls in, including a specific and detailed path to solving the problem that is currently alerting. With the Transmogrifier, this feature is only a few clicks away and I am speaking to our team on this on a regular basis in order to improve the team’s efficiency bringing newer folks up to speed faster. The mobile app interface is outstanding and is constantly improved based upon user feedback.

Nachteile:

The only Con that I have discovered is the time it takes to weed through the alerts that you setup. The robustness of what alerts can be programmed in to be registered is an ongoing process as new systems get added.

VictorOps FTW

Vorteile:

I enjoy the clean direct design. It makes it very easy to use. It was by far the easiest and most simple to setup when comparing other products.

Nachteile:

The main negatives of VictorOps are that it doesn't seem to be as flexible as some of the competition in terms of how alerts can be accepted, there are some strict guidelines. I would also like to see some improvements made on the alerting - for instance to put in a delay. I also don't like that some transmognifier features are only available in enterprise.

Save time and energy with VictorOps

Kommentare: VictorOps allows you to consolidate alerts coming from multiple platforms, into one easy to read interface (and hand mobile app). I would recommend this service for anyone who is responsible for maintaining services and receives alerts from multiple platforms.

Vorteile:

This has made all the difference for our OnCall

Nachteile:

Haven't quite figured out how to make VictorOps resolve our issues for us ;-)

Splunk On-Call Review

Kommentare: My overall experience with Splunk On-Call is too good. It is easy to use and help organisation to manage the on-call support engineer to handle the incident get raise in the production environment.

Vorteile:

The most impactful feature of this software is that we can easily schedule the on-call support engineer for our production environment. So, the team can easily reach out to the available engineer. and also help to let us know what incidents get raised in a production environment. It helps organisations to manage the support members. Engineers will get notifications about their on-call week and incident get triggered in the env. It is easy to use.

Nachteile:

As of now, I don't have anything which I don't like about this software.

Manage the firehose of information

Kommentare: Overall, VictorOps has significantly improved our signal to noise ratio by letting us pay attention just to the important notifications (like the site being down) rather than the many many other logs we gather.

Vorteile:

I love that it lets you customize contact schedules between different users in a company so granularly. This is a new product for us and we are a small company so we need to be careful about easing people into these types of dev-ops schedules and VictorOps allows us to do so.

Nachteile:

In certain places of the UI, it doesn't make sense the order in which you have to do certain actions like adding users vs creating a schedule. Ultimately we figured it out but it's very strict in some places about how exactly you go about organizing your company's scheduler.

VictorOps is our Alerting Funnel

Kommentare: VictorOps has helped us keep all of our alerts in a single place. Don't underestimate how much time you save having an authoritative source for all of your alerting. From VictorOps you can then dole out the alerts to only the teams and therefore the on call member that needs to see the alert. On call member doesn't take care of the issue? You can then automate the process of alerting more people in the team, it's all up to you.

Vorteile:

Simple when you want it to be, more advanced when you need it to be.

Nachteile:

Could work better with non-standard schedules. Not every group we work with does on call in standard follow the sun or one week on type of scheduling. Some do on-call in spurts of 4 or 3 hours depending on events.

VictorOps is a great tool for alert routing and escalation!

Kommentare: VictorOps is a great application to funnel and route alerts from all your systems in one area. It has scheduling, so any team member is never always on call. The iOS app is a great tool for acknowledging alerts.

Vorteile:

alert routing and team scheduling are great features. annotations for leaving notes on how to handle an alert for new team members.

Nachteile:

not sure I can think of one at this time

VictorOps is helping us to build our Sustaining organization

Kommentare: Excellent service for tracking server outages as they happen

Vorteile:

I like the best is the simplicity to set it up and to integrate with other tools, as well as the versatility to create different teams with different escalation schedules and paging policies.

Nachteile:

The only thing thus far I have a slight issue with is not having a tabbed view of environments, Since the calls use text to speech, it's sometimes difficult to hear the name of the server which has an outage, though once aware you can just check the app.

On-call sucks less now

Kommentare: We started using VictorOps six months ago, and now it's almost a pleasure being on call. Alerts come from a single source now, and only when you're on your rotation. (Before everyone on the team was getting a mix of text messages & emails 24x7) The timeline gives a good summary of events, and makes handing off on-call responsibilities a breeze.

Vorteile:

The incident timeline is a great way to view events. Mobile apps are great on both iOS & Android.

Nachteile:

Transformation system (transmogifier) can be pretty confusing at times. Some features seem to only exist in mobile.

Great on-call management for small/medium teams

Kommentare: I can't really say much about this tool other than: it works, and it works well. Having tried a competitors offering, we found VictorOps to be the right balance of flexible, well-priced, and reliable. It has become an integral part of our operations. We use it with HipChat and find that it makes our incident response and tracking painless. And by painless, I mean it stays out of your way and lets you focus on the problem at hand.

Vorteile:

Great price/value Easy workflows Incredible integrations

Plain Alerting

Kommentare: This is a really good tool to integrate into your tech stack if you need an alert filter, which you probably do. The down side is that it's very basic and routing alerts takes a degree of skill deploying regex or matching conditions. The cost point is very amicable for a decent filtering tool, but the tool itself has a lot of room for polish which could be a detriment to less technical departments.

Vorteile:

The software does a good job of the basics. It seemed easy enough to give out to the whole team and we seem to get the needed functionality out of it. The reports are straightforward and easy to identify the problem areas. Integrations are via API so any noise generator that can make an HTTP POST in theory could be integrated.

Nachteile:

5,000 total alerts per month maximum which may sound like a lot, but a growing enterprise company may find that total to be limiting. The UI is very basic, possibly by design or my limited permissions but in a lot of ways it feels hollow and lacks any real character if you have to use it daily. The mobile application works well, and again keep things organized, but underwhelms visually and doesn't captivate the user in any way.

VictorOps to reduce downtime and increase uptime

Kommentare: Excellent service for the notification of incidents in the system. It is possible to set different rules and schedule on a group of users: DevOps, Managers. ViсtorOps team quickly implement feature requests - thank you very much!

Vorteile:

Good Android application to manage/acknowledge incidents. East integration with Zabbix. Nice community and documentation.

Nachteile:

The application for Android has no buttons on / off for Maintenance mode.

VictorOps For the Win

Kommentare: Best move the company has made to make our on-call rotation be easier. Amazing integration and support. The team is also very responsive to questions.

Vorteile:

Ease of integration. Quick response on bugs. Proximity to their headquarters. The People!